The most common AI ROI conversation we have starts with a client telling us a new AI tool "feels faster" or "the team loves it," and ends with us asking for a number a CFO would accept. Vibes don't survive a budget review. Here's the framework that does.
Start with a baseline captured before the AI system existed - time-to-resolution, error rate, cost per unit of work, whatever the workflow's actual bottleneck metric is. Without a real baseline, every improvement claim is unfalsifiable. Then measure the same metric post-deployment on a comparable sample, not cherry-picked cases. If a support-triage model claims to cut response time, measure it across the full ticket volume, not just the tickets it handled well.
Separate throughput gains from quality gains, because they trade off against each other more often than vendors admit. An AI system that halves review time but doubles the error rate hasn't actually saved money once you account for rework and the cost of the errors that get through. We build both into the same dashboard so a client can see the real trade curve, not just the headline speed number.
Finally, account for the fully-loaded cost, not just the API bill - the time an engineer spends maintaining the prompt/retrieval pipeline, the cost of periodic re-evaluation as the underlying model changes, and the incident cost of unexpected failures. Clients who do this exercise properly usually find the AI system was still a clear win - but the number that survives the exercise is a fraction of the number the initial pitch promised, and that's the number worth trusting.
