Decoding the AI ROI Puzzle: A Multifaceted Approach

To shed light on this challenge, we spoke with technology leaders across industries to understand how they are evaluating AI performance. The methods range from straightforward comparisons against human benchmarks to sophisticated frameworks incorporating cultural shifts, cost modeling, and advanced value calculations.

The Baseline: Can AI Outperform Humans?

At the heart of nearly every AI metric lies a fundamental question: how does AI perform on a given task compared to a human counterpart? Branz emphasizes applying the same evaluation criteria to AI as those used for human performance. “AI can accelerate work, but faster doesn’t equal ROI,” she explains. “We measure it the same way we measure humans – by observing if it generates real results like traffic, qualified leads, and conversions. A KPI we’ve found useful is cost per qualified result, which measures how much less it costs to achieve a tangible outcome.”

The key, Branz adds, is contextual comparison. “We try to isolate the impact of AI by running A/B tests between AI-generated and human-created content. For example, when testing AI-generated copy or keyword groups, we track the same KPIs – traffic, engagement, and conversions – and compare the results to those achieved by humans alone. We view AI performance as a directional metric, not an absolute one. It’s incredibly useful for optimization, but it’s not the final judgment.”

Marc-Aurele Legoux, founder of an organic digital marketing agency, offers a more direct assessment: “Can AI do this better than a human? If so, game on. If not, it’s not worth the investment. For instance, we implemented an AI-powered chatbot for a luxury travel client, and it generated an additional €70,000 in revenue from a single booking.”

Legoux’s KPIs are straightforward: “Did the chatbot initiate the lead? Yes. Did that lead convert into a customer? Yes. Thank you, AI chatbot.” He then compares AI-generated results to human performance over a defined period. If AI matches or exceeds human benchmarks, it’s considered a success.

However, establishing valid comparisons, controlling external factors, and attributing results solely to AI is often far more complicated in practice.

Beyond Speed: Time, Accuracy, and Value Creation

The most tangible aspect of AI ROI often relates to time and productivity gains. John Atalla, CEO of Transformativ, calls this “productivity amplification” – time saved and capacity unlocked, measured by the duration of processes or tasks. “In our initial projects, our KPIs were fairly limited,” Atalla admits. “Over time, we saw improvements in decision quality, customer experience, and even employee engagement, all with measurable financial impact.”

This led Atalla to develop a framework based on three perspectives: productivity, accuracy, and “value realization velocity” – the speed at which business benefits are observed, measured by payback period or the proportion of benefits realized within the first 90 days.

At Wolters Kluwer, Aoife May, Director of Product Management, explains that her teams compare manual and AI-assisted work to calculate time and cost differences. “We attribute estimated times to tasks like manual legal research and include an average hourly cost for lawyers to identify human effort. We then do the same for AI,” she says. Clients, she notes, “reduce time spent on obligation research by up to 60%.”

But time savings aren’t the whole story. Atalla’s second perspective, accuracy, measures gains from reduced errors, rework, and exceptions, translating to lower costs and improved customer experiences.

Adrian Dunkley, CEO of StarApple AI, frames the financial perspective at a higher level: “There are three categories that always matter: efficiency gains, customer spend, and overall ROI.” He tracks “how much money has been saved thanks to AI and how much more has been generated without increasing costs.” Dunkley’s research lab, Section 9, tracks AI’s specific contribution through “impact chaining,” mapping each process to its downstream value to create a “pre-AI ROI expectation.”

Tom Poutasse, Director of Content Management at Wolters Kluwer, uses a similar approach to demonstrate where automation adds efficiency and where human judgment provides essential precision.

Getting the Calculations Right: Baselines, Attribution, and Cost

Calculating ROI begins with clean baselines and ends with understanding how AI redefines the cost of doing business.

Salome Mikadze, co-founder of Movadex, recommends rethinking what’s measured: “I tell executives to stop asking ‘What’s the model’s accuracy?’ and start asking ‘What changed in the business once it was implemented?’”

Mikadze’s team integrates these comparisons into every deployment. “We establish a pre-AI process baseline and then execute controlled implementations to ensure each metric has a clean counterfactual,” she explains. This might involve tracking first response and resolution times in customer service, code change delivery times in engineering, or success rates and content cycles in sales. But, she stresses, all these metrics include time to value, active user adoption, and task completion without human intervention, because an unused model has zero ROI.

More broadly, measuring ROI also means understanding the true cost of AI. Michael Mansard, Principal Director of the Subscribed Institute at Zuora, points out that AI disrupts the economic model IT has taken for granted since the dawn of SaaS: “Traditional SaaS is expensive to build, but its marginal costs are nearly zero. AI, however, is cheap to develop but generates high variable operating costs. These shifts challenge seat- or feature-based pricing models, because they fail when value is measured by what an AI agent achieves, not by how many people log in.”

However, baselines can become blurred when humans and AI share the same workflow, a challenge that led Poutasse’s team at Wolters Kluwer to completely rethink attribution. “We knew from the start that both AI and human experts brought value, but in different ways. So simply saying ‘AI did this’ or ‘humans did that’ wasn’t accurate.” The solution was to implement a tagging framework classifying each step as machine-generated, human-verified, or human-enhanced. This accurately measures where automation adds efficiency and where human judgment provides essential context.

5 Recommendations for CIOs Measuring AI ROI

1. Don’t focus solely on model accuracy. The real measure is the change in the business. Before deploying any AI system, establish a pre-AI process baseline and run controlled implementations to ensure each metric has a clean counterfactual for comparison.
2. Recognize that AI changes the rules of the traditional SaaS economy. Traditional IT has low marginal costs, but AI involves high and variable operating costs. Abandon simple per-seat pricing; explore usage- or outcome-based pricing, where value is directly linked to the AI agent’s impact (e.g., cost per resolution).
3. AI success isn’t measured solely by gross profits, but by reliability. Model the total cost of ownership (TCO) and adjust expected benefits considering security and reliability signals. This includes metrics like hallucination rates, safeguard interventions, overwrite rates, and model drift.
4. AI and humans share workflows; attributing results solely to “AI” is inaccurate. Implement a tagging framework marking each step as machine-generated, human-verified, or human-enhanced to accurately measure where automation adds efficiency and where human judgment provides essential context.
5. Long-term success depends on employee adoption and trust. Measure “soft” ROI in the early stages – employee satisfaction, usage rates, self-reported productivity. These indicators drive engagement, fueling a virtuous cycle of adoption that unlocks “hard” ROI later.

Mansard notes that some companies are experimenting with results-based or percentage-of-savings pricing. There’s no one-size-fits-all model: “Many are opting for usage- or outcome-based fees, where value is tied to the AI agent’s impact.”

Frequently Asked Questions About AI ROI

1. What is the most important factor when calculating AI ROI?

   The most crucial factor isn’t the technical accuracy of the AI model, but rather the demonstrable change in your business outcomes after implementation. Focus on measuring tangible results like increased revenue, reduced costs, or improved customer satisfaction.

2. How does AI impact traditional SaaS pricing models?

   AI disrupts traditional SaaS pricing because its value is tied to the impact of the AI agent, not simply the number of users. Consider usage-based or outcome-based pricing models that align costs with the value delivered.

3. Why is it important to measure the “soft” ROI of AI?

   “Soft” ROI, such as employee satisfaction and adoption rates, are leading indicators of long-term success. If employees don’t trust or embrace AI, it won’t deliver its full potential.

4. What is “impact chaining” and how does it help with AI ROI measurement?

   Impact chaining maps each process to its downstream value, creating a “pre-AI ROI expectation.” This helps you understand the specific contribution of AI to the overall business value chain.

5. How can organizations ensure accurate attribution of results when humans and AI collaborate?

   Implement a tagging framework that classifies each step in a workflow as machine-generated, human-verified, or human-enhanced. This provides a more precise understanding of where automation adds efficiency and where human judgment is essential.