Optimisation landscape
Research date: 2026-10-22
Purpose: choose the right engineering intervention for a leadership decision
Evidence: primary documentation and research, with explicit analyst interpretation
Core conclusion
Optimisation should minimise the total cost of delivering a specified outcome subject to quality, latency, access and reliability requirements. The useful order of investigation is usually demand and workflow, model fit, repeated work, serving efficiency, then hosting economics. This is a diagnostic order proposed for this session, not a universal ranking of savings.
A technical improvement becomes a financial saving only when it reduces a paid resource, avoids a credible future purchase, or releases capacity that actually serves useful demand. Keep these outcomes separate.
1. Define the workload before choosing a lever
| Workload | Cost-sensitive design questions | Protected outcome |
|---|---|---|
| Offline product-data enrichment | Can rules handle known fields? Can changed records alone be processed? Can requests wait for a batch? | Correct fields accepted for the right product, market and language |
| Interactive technical-information assistant | How much evidence is retrieved? How often does it abstain or escalate? Can common questions be reused safely? | Correct, source-supported answer within the service target |
| Internal coding/research agents | How many tools, retries, parallel branches and context replays happen per completed task? | Accepted artifact or resolved task, with review effort included |
| Custom voice, vision or multimodal system | Which stage dominates total delay or memory? Are peak concurrency and warm-up measured? | End-to-end usable output under the required conditions |
The first two are proposed Knauf examples, not descriptions of Knauf’s current AI implementations.
2. The optimisation decision matrix
The “first experiment” and risks are analysis. Source IDs establish the underlying mechanisms.
| Lever | When to investigate | First experiment | Main limitation | Evidence |
|---|---|---|---|---|
| Remove unnecessary AI calls | Repeated formatting, lookups or constrained transformations | Replace a narrow step with a rule, query or template | Rules need maintenance and coverage checks | O02 |
| Simplify workflow | A task fans out into calls with little measured quality gain | Remove one redundant branch and compare accepted outputs | Some hard tasks benefit from added reasoning or agents | O01, O07 |
| Control agent execution | Cost has a long tail or grows after failures | Bound steps, retries, tool access and per-task spend with a useful fallback | Tight caps can lower completion rates | O01, O07 |
| Reduce irrelevant context | Long repeated inputs, noisy retrieval or tool results | Filter context while preserving required evidence and permissions | Pruning can remove the decisive document or exception | O02 |
| Shorten unnecessary output | Generated prose exceeds the actual field or user need | Match response length to the task’s acceptance criteria | Artificial truncation can produce unusable output | O02 |
| Select a smaller/specialised model | Repetitive task with measurable acceptance criteria | Evaluate a lower-cost candidate on representative cases | Generic benchmark performance does not establish domain fitness | O08 |
| Route or escalate | Task difficulty varies and a strong fallback exists | Compare a routing rule with the single-model baseline including router cost | Routing errors, drift and fallback calls can erase gains | O03 |
| Reuse prompt prefixes | Stable instructions or reference text recur | Measure actual cache reads, writes and realised bill | Prefix reuse does not reuse generated answers | O04, O05 |
| Cache completed answers | Identical questions recur against stable, authorised information | Cache with document/version/permission keys and expiry | Wrong reuse can expose data or serve stale advice | Analyst design, O04 distinguishes prefix caching |
| Use asynchronous batch | Work has a loose delivery deadline | Run an eligible non-interactive job and include expiry/retry handling | Discount trades scheduling freedom and response immediacy | O06 |
| Fine-tune or distil | Stable, high-volume task has suitable training examples | Small scoped pilot including data, training, evaluation and upkeep | Setup cost, drift, license restrictions and quality loss | O08 |
| Quantise | Model weights dominate memory and a supported execution path exists | Benchmark the exact hardware/model/runtime combination | Smaller weights do not guarantee proportional speed or lower bill | O09 |
| Tune batching and scheduling | Demand can share a serving pool | Replay arrivals and record useful throughput plus tail latency | Waiting to form batches can violate latency targets | O10, O11 |
| Optimise the system bottleneck | A custom pipeline has measured hot stages | Profile end to end, change one stage, remeasure whole pipeline | Faster local kernels may barely change total performance | I04, I05 |
| Speculative decoding | A supported model/runtime has suitable decode-heavy traffic | Compare under the actual concurrency and output mix | Drafter cost and acceptance rate matter; gains vary | O12 |
| Change hosting or capacity | Spend is material and demand/model requirements are understood | Compare a full quality-qualified TCO with demand sensitivity | Idle capacity, support, resilience and migrations can dominate | Cost-economics research |
3. Important distinctions for the talk
Prompt caching, answer caching and warm processes
Prompt caching reuses processing of a shared prefix. It still generates the answer. vLLM specifically notes that this addresses prefill and does not accelerate the generation phase. Answer caching reuses a previous result and requires a separate correctness and access design. Keeping a process warm avoids loading/setup work but may continue to incur infrastructure charges while idle. O04
OpenAI’s current cache rules vary by model generation and retention setting, including cache-write accounting. Use observed paid usage rather than assuming all repeated input earns the same discount. O05
Routing and fallback
RouteLLM demonstrates that routing is a real engineering method, but its results depend on training data, model pair and evaluation. Its authors also report weak out-of-domain results for some routers before targeted data augmentation. The session should explain the principle without promising a transferred percentage. O03
API batch and online batching
A vendor Batch API defers work under a service contract. Dynamic batching in a self-hosted server combines concurrent requests to use hardware efficiently. They are distinct mechanisms. NVIDIA’s scheduler exposes queue-delay controls, and vLLM tuning describes trade-offs among prompt processing, decode and throughput. O06, O10, O11
Technical efficiency and money saved
If weights shrink but the same GPU remains allocated for the same hours, the cash bill may remain unchanged. If the freed memory enables a smaller instance or useful co-resident work, a financial benefit becomes plausible and measurable. Count operator effort and new reliability risks. This is cost-accounting analysis rather than a vendor claim.
4. Quality gates before cheaper deployment
Use the same held-out task set for the baseline and candidate. Include hard cases, languages, product variants and missing evidence. Separate routine quality metrics from critical errors that cannot be averaged away. Use domain review where a fluent answer could be technically wrong. Compare completion, correction effort, latency distribution and total cost. Keep a rollback path.
The evaluation principle has primary support in official guidance. The exact Knauf acceptance criteria and sample sizes must come from the workload owner, not this dossier. O08
5. What deserves main-stage time
Give approximately four minutes to workflow and model choices, using one example. Give two minutes to hosting economics and one minute to a measured lesson about system bottlenecks/quality. Keep framework names, quantisation formats, speculative decoding and provider price lists in research or Q&A. The audience needs to know which question to ask its engineering team and how to judge the result.
6. Claims to avoid
- Every enterprise can cut AI spend by a fixed percentage.
- The smallest model or cheapest token rate yields the cheapest usable result.
- Model self-reported confidence is a calibrated routing policy.
- A cache hit is guaranteed because the same document was supplied.
- More parallel agents automatically mean greater business productivity.
- Lower GPU memory directly equals lower financial cost.
- A fast component benchmark establishes an end-to-end service target.
- A working prototype establishes product release, operational resilience or domain quality.