Prove value with matched outcomes and a credible counterfactual
Recommendation: Begin with a narrow business task, record its baseline, then measure whether the intervention improves delivery and economics at equivalent quality. Distinguish self-reported productivity, measured task improvement and realised financial benefit.
1. Use an evidence ladder
| Evidence | What it supports | What it does not establish |
|---|---|---|
| Product documentation | A capability or constraint exists | Customer benefit or expected savings |
| Provider case/benchmark | Result in the reported setup | Independent replication or general transfer |
| Systems research | Measured technical mechanism under test conditions | Corporate ROI or current-runtime superiority |
| Workplace experiment | Effect on studied workers/tasks | Effect on every profession or newer tools |
| Survey | Respondents’ experience and perceptions | Causal financial impact |
| Customer pilot with matched baseline | Local task and economic effect | Unlimited scalability without further validation |
Use the strongest relevant evidence available, not the most impressive percentage.
2. Four cases suitable for leadership discussion
Case A: customer-support assistance can improve throughput, unevenly
Brynjolfsson, Li and Raymond’s revised study follows 5,172 support agents and reports 15% more issues resolved per hour on average. Gains differ by experience; the most experienced group saw small speed gains and small quality declines. The inspected arXiv revision is November 2024; the journal version appeared in 2025. Earlier working-paper summaries used different sample/count figures, so keep the version consistent. S25
Our lesson: Evaluate the intended user group and task. Measure quality and throughput together; released capacity needs an operational use before it becomes economic value.
Case B: measured developer effects can change with the tools and study design
METR’s February 2026 update revisits its early-2025 finding of 19% longer task completion. Its later experiment has selection and measurement problems; the authors explicitly say it provides an unreliable signal of the current effect. Reported later estimates are uncertain, not definitive proof of present-day speedup. S26
Our lesson: Use task-specific measurement and revisit it as tools change. Do not use the historical slowdown as a blanket October 2026 claim about AI coding.
Case C: verification can make smaller models economically useful
OSCAR, submitted 1 October 2026, reports 95–100% accuracy on five operations-research benchmark problems using iterative verification and two small open-weight models. Comparison coding agents incurred 3.1× and 5.8× average token costs in the reported runs. The result includes problem-specific simulator preparation and a small experimental scope; token cost excludes a complete enterprise lifecycle comparison. S20
Our lesson: Investigate testable constraints and verification before defaulting to a costly model. Transfer to Knauf would require its own labelled cases and an economic account of verification setup.
Case D: production profiling exposes workload-specific inefficiency
Herschel’s September 2026 preprint reports production profiling and deployable improvements in Alibaba’s serving environment. It supports investigating the real workload rather than treating a benchmark as the final optimisation result. This is an author-reported production systems study, not independently verified customer ROI. S21
Our lesson: Preserve request conditions when testing improvements and include ongoing maintenance in the economic decision.
3. Use survey evidence as context, not a savings forecast
McKinsey’s State of AI survey was published 25 August 2026, fielded 4 May–8 June, with 1,719 participants across 97 countries. It reports 37% attributing positive EBIT impact to AI and roughly 20% saying operating costs constrain usage. These are respondent reports, not audited financial outcomes or evidence about Knauf. S24
Our interpretation: A leadership discussion should connect adoption, workflow change and expenditure with demonstrable outcomes. Cutting inference cost alone may not resolve weak value realisation.
4. Worked illustration: a cheaper model can increase total cost
All numbers below are hypothetical. Assume 100,000 documents/month, 100% reviewed to a common acceptance standard, and identical final quality. Baseline AI cost is €0.020/document; review averages 60 seconds at €36/hour.
| Scenario | AI cost/document | Review time/document | Monthly AI + review cost |
|---|---|---|---|
| Baseline | €0.020 | 60 seconds | €62,000 |
| Cheaper model, more review | €0.010 | 72 seconds | €73,000 |
| Better workflow, same AI cost | €0.020 | 45 seconds | €47,000 |
Calculation: review costs €0.01/second. Thus the respective per-document totals are €0.620, €0.730 and €0.470.
The middle option halves the model bill and raises the combined task cost by €11,000/month. The third releases 416.7 review hours/month relative to baseline, valued at €15,000 at the stated rate. That is a capacity valuation; it is not automatically a €15,000 cash saving.
If human labour is fixed in the decision period, the accounting invoice may stay unchanged. Value may instead appear as avoided overtime, reduced backlog, more completed work or deferred hiring. Include fixed operating and implementation costs before claiming net ROI.
5. Define the business case before the pilot
Proposed decision framework:
incremental_net_benefit
= attributable_realised_benefit - incremental_total_cost
For a defined period, ROI can be expressed as incremental net benefit divided by the relevant incremental investment/cost. State the denominator and treatment of recurring costs. Calculate payback only where benefit timing and cash effects are defensible. Use ranges if benefit attribution is uncertain.
Separate:
- Actual expenditure reductions.
- Cost avoidance against a documented baseline plan.
- Capacity released and where it is redeployed.
- Revenue/contribution improvements with attribution assumptions.
- Quality or risk benefits measured in their own units.
Do not monetise the same hours twice as both payroll savings and increased output.
6. A pilot that leadership can act on
Proposed initial scope: one document workflow or internal knowledge task, a named owner, representative real cases and an agreed acceptance standard.
- Record baseline effort, errors, task volume and service level.
- Instrument the existing workflow before changing it.
- Evaluate candidate changes on a held-out set, including difficult and multilingual cases.
- Compare matched workloads using randomisation or a credible before/after design with explicit confounders.
- Run a limited production canary and review exceptions.
- Decide whether to scale using net benefit, implementation effort, service capacity and operating ownership.
For knowledge assistants, include answer correctness, evidence support and escalation. For extraction, include field-level accuracy and downstream correction. For visual inspection, include false accepts/rejects and process impact. These are prospective use-case examples, not claims about Knauf’s deployed systems.
7. An engineering-output proxy is a diagnostic, not cash ROI
Navigara’s Q2 study is observational, does not quantify AI attribution and documents historical rescoring. S44 Use it to discuss measurement discipline, not to forecast savings. Freeze scope, cohort, scoring version and snapshots before comparisons. Measure the full path through review and release, rather than generation alone. S45
Retain business-outcome and quality measures alongside throughput. DORA’s current official framework has five delivery metrics; a vendor output score does not replace them. S48 A capacity benefit divided by cost is a benefit/cost ratio, not net ROI. Realised cash, avoided expenditure and redeployed capacity require separate reporting. The arithmetic and claim checks are in chapter 10 [local reference omitted].
8. Use Halbach evidence precisely
A latency improvement can support a service-performance claim. It does not by itself establish reduced cost, increased throughput or better commercial outcomes. Before using an internal case, confirm workload, measurement boundaries, hardware, model/version, concurrency, quality, paid capacity and permission to disclose partner information.
Presentation implication: choose one external empirical case and one approved Halbach case. Use the document illustration if a financial explanation is needed; label it visibly. Reserve preprint percentages for an appendix.