Alaaeddine Hammoudi

Writing / article

Measuring AI: what to count, and how

AI transforms each organisation differently, so its value has to be measured, not assumed. Four indicators for every use case, one guardrail to protect what makes the business different, and how to measure both honestly.

Author
Alaaeddine Hammoudi
Published
Reading
5 min

In short

  • AI changes each organisation differently, so its value has to be measured, not assumed.
  • Stop counting activity. Track four indicators per use case: value, quality, freed capacity and dependency.
  • Time saved is only potential value. It becomes real when it is reinvested.
  • Add one guardrail: no use case may erode your business alpha, the judgement that sets you apart.

Same AI, different effects

IKEA’s bot, Billie, has handled about 47% of customer queries since 2021. Ingka Group didn’t cut jobs. It retrained around 8,500 call-centre workers, mostly as remote interior-design advisers, and that channel brought in €1.3bn of sales in 2022, about 3.3% of the total. Billie predates generative AI, and not every sale was new. But the value came from what people did with the freed time, not from the bot itself.

Microsoft leased AI compute from Oracle in 2023 to move fast. According to Bloomberg, citing unnamed sources, it is now letting the lease lapse because the servers’ previous-generation Nvidia chips no longer meet its needs. There, the value was speed, plus the freedom to walk away while the lessor kept the obsolescence risk.

One company kept its people and changed their work. The other rented capacity and is handing it back. Each choice fits what was being transformed, and no adoption figure would have told you which was right.

Count value, not activity

In METR’s 2025 trial, experienced developers were 19% slower with AI but believed they were 20% faster. Organisations make the same mistake at scale: licences, prompts and satisfaction scores are easy to count, so dashboards fill with them. Value needs a baseline, a price and an owner.

Four indicators

1. Value: the euro/AI ratio. Each use case has one countable unit, such as a request answered or a document processed. Count how many are treated per period, and price each one at what it would cost without AI:

euro/AI = (requests treated without rework per period × € per request) ÷ (inference + review + corrections + run over that period)

For example, an assistant that treats 10,000 requests a month, each worth €4 of staff time, at a full cost of €25,000 a month, scores 1.6 (illustrative figures). Where the value is risk avoided, as in compliance checks, price each request at the expected cost of the incident it prevents. Above 1, the use case can pay for itself. It is potential value, not yet realised.

2. Quality: the rework rate. This is the share of outputs someone has to correct, redo or escalate. Without it, volume hides poor work.

3. Freed capacity: where the saved time goes. This is what turns potential value into real value. Saved hours count only if people are redeployed, growth is absorbed without hiring, or the cost comes out. Write down the plan before launch, and check it six months later.

4. Dependency: keep or lease. Keep what appreciates, lease what depreciates. Hardware and models lose value with every release, so lease them. Know-how, customer relationships and proprietary data gain value, so keep them. For most companies, the real lock-in is a SaaS tool that holds their process logic and data, or a provider that sets its own prices and retirement dates. Record the exit cost for each use case.

The guardrail: business alpha

In finance, alpha is the return earned above the market through skill. A business has its own: the judgement and know-how competitors can’t buy. Everyone now rents the same models, so whatever you delegate to them, competitors can match. Your edge is what the model doesn’t know.

Delegating execution, such as sorting, drafting or routing, leaves that edge intact. Delegating judgement, such as diagnosing, deciding or explaining, erodes it. Experienced endoscopists who got used to an AI assistant detected fewer adenomas without it, down from 28.4% to 22.4% (Lancet Gastroenterology & Hepatology). Among 319 knowledge workers, more confidence in AI went with less critical thinking (Microsoft Research and Carnegie Mellon).

Lost alpha never shows up in the ratio, which is why it is a guardrail and not a KPI. For each use case, name the judgement that differentiates you. Keep a named person able to exercise it without AI, and test that with periodic AI-off exercises. A use case that breaches the guardrail is reworked or stopped, however good its ratio.

The scorecard

IndicatorMeasureAct when
Valueeuro/AI ratio, per quarterBelow 1, or falling for two quarters
QualityRework rateAbove the pre-AI baseline
Freed capacityHours freed, and where they wentHours saved but not reinvested
DependencyKept or leased, exit costExit cost growing faster than value
GuardrailBusiness alpha held by a named personBreached: rework or stop

Five rules

  1. No baseline, no production. Without a “before” figure, any gain is a feeling.
  2. Logs over surveys. Satisfaction shows people like the tool, not that it delivers.
  3. Finance prices the request, once for everyone, so no team picks a flattering number.
  4. One owner per indicator, reviewed every quarter.
  5. Report adoption separately. It’s context, not value.

AI will keep getting cheaper and faster. The organisations that measure it will know where it pays and what it must never replace. The others will only know how it feels.

Sources