|
AI, Applied
Agents are moving into real work. "Measurable ROI" doesn't always mean what it sounds like.
AI, Applied
What changed. What matters. What to do about it.
Issue 04 · Fortnightly · 5-minute read
|
|
|
AI agents are moving into real work — but “measurable ROI” doesn't always tell you what was actually measured.
Anthropic's 2026 State of AI Agents Report says 80% of organisations are seeing measurable ROI from AI agents. It's already being read as proof that agents have moved from experiment to everyday enterprise work. But what does “measurable ROI” actually mean here?
The 80% comes from a survey of 500-plus technical leaders, run by an independent research firm in late 2025 and commissioned by Anthropic. Respondents were asked whether they agreed their AI investment “has already delivered measurable financial impact.” That's a perception from technology staff — not an audited financial result, and not a question put to finance.
The report's own case studies back this up. Novo Nordisk cut clinical documentation from ten-plus weeks to ten minutes. Doctolib shipped features 40% faster. L'Oréal lifted analytics accuracy from 90% to 99.9%. N26 hit 70% automation on targeted processes. These are meaningful reported gains — time, speed, accuracy, capacity. But none of that is automatically the same as a financial return. Where a dollar figure does appear in the report, it's usually the vendor's own revenue, not the featured business's result: one case study cites a start-up's $40 million in annual recurring revenue, another a $1 million licensing deal.
Source: Anthropic →
|
|
The claim
“80% of companies say AI agents deliver measurable ROI.”
|
→ |
The real question
“Measured how, by whom, against what baseline?”
|
|
|
|
Every AI result sits somewhere on a five-step ladder. Each step is real evidence — just not the same kind:
|
1 |
Activity / adoption — is anyone using it? |
2 |
Time or capacity saved — did it save time or free up capacity? |
3 |
Estimated economic value — what's that saving estimated to be worth? A calculation, not a measurement. |
4 |
Converted business value — did the business actually use that benefit: more output, better service, sharper decisions, lower risk? |
5 |
Realised financial benefit — did a financial result show up: lower cost, more revenue, avoided loss? |
|
|
|
Much of Anthropic's own case-study evidence sits on rungs one and two: adoption, time saved, faster processes. Real results — just not the same as a realised financial return.
MIT's NANDA research paints a very different picture. After reviewing 300+ deployments, interviewing 52 organisations and surveying 153 senior leaders, it found that 95% showed no measurable profit-and-loss impact. The research is preliminary and, like Anthropic's, relies partly on what organisations report rather than audited financials. So the headline numbers — 80% versus 5% — aren't directly comparable. And that's exactly the point: different studies are calling different things “AI success.”
Source: MIT NANDA →
Australia's own guidance makes the same point. The National AI Centre's approach: measure the time a task takes now, measure it with AI, then multiply the difference by the cost of that time. That's a legitimate way to estimate value. But there's a catch — time saved only becomes business value if it's put to use: serving customers, improving quality, growing the business. And not all value needs to show up as a dollar figure. Better decisions, higher quality and lower risk count too. The point is simple: know which kind of result you actually have.
Source: National AI Centre →
The shift isn't that agents don't work — the evidence above says plenty of them do. It's that “ROI” can be used to describe very different kinds of value, without always making clear which one. Before anyone tells you an AI initiative has delivered ROI, ask what was actually measured.
|
|
|
|
Three actions, not a checklist
|
1
|
Find your own rung.
For every AI agent running in your business, ask which of the five levels its result actually sits on — before calling it a return.
|
|
2
|
Set the baseline before the next investment, not after.
Measure what a process costs, takes or gets wrong today. Without that number, nothing you compare it to later can be called measured.
|
|
3
|
Ask what the gain actually became.
Time saved is useful evidence. The next question is what it became: more output, better service, sharper decisions or lower cost. Check whether that happened — or is still assumed.
|
|
|
|
|
|
Even the executives closest to AI spend often can't say it's paying off
Gartner's research on CIO planning for 2027 finds just 13% of CIOs report significant value from AI tools so far. Only around a third expect to hit their cost-reduction targets. More than half expect AI to raise — not lower — total cost of ownership, as agentic workloads push consumption costs up. Even inside IT departments, the numbers on AI's payoff are still thin.
Source: Gartner →
|
|
|
|
|
|
|
One Question For Your Business
Which AI use case in your business could you prove created value — not just saved time?
|
|
|
That's it for this edition — short on purpose. If one of these changes what you do this week, it did its job.
|
|
Mia Pursehouse · Business Analyst specialising in AI
|
|
|
AI, Applied is an executive briefing for leaders who don't have time to follow AI but need to know what it means for their business.
|
|
AI transparency: AI tools assisted with research and drafting. Final selection, analysis and editorial review by Mia Pursehouse.
|
|
Read past issues
·
Manage preferences
·
Unsubscribe
Mia Pursehouse, Buttaba, NSW 2283
Sent to Reader
|
|
|
|
