Choose the intervention that improves accepted output per euro
Recommendation: Identify the dominant cost and service bottleneck before changing the system. Evaluate workflow, model, serving and capacity changes as separate hypotheses, then test their combined effect.
Decision rule: minimise relevant cost at an agreed quality threshold and latency/reliability objective. A reduction in one component is valuable only if it survives the end-to-end comparison and justifies implementation effort.
1. Map interventions by mechanism and prerequisites
This table is our synthesis of documented techniques and operating trade-offs, not a ranking of universal savings potential.
| Intervention | Mechanism | Best candidate | Main test or limitation |
|---|---|---|---|
| Remove unnecessary calls/steps | Avoid computation and external charges | Repeated checks, redundant orchestration | Verify the removed step was not protecting quality |
| Bound agent execution | Limits steps, retries, branches and elapsed time | Variable-length autonomous workflows | Completion rate at the limit and escalation behaviour |
| Improve retrieval/context selection | Send relevant evidence with less repeated context | Document-heavy workflows | Missing evidence, recall and citation accuracy |
| Response caching | Reuse a valid prior result | Stable repeated tasks | Freshness, authorisation and invalidation |
| Prompt/prefix caching | Reuse processing of a matching prefix | Repeated long instructions or documents | Reuse frequency, expiry and cache-write economics |
| Smaller or specialised model | Reduce per-call compute or API rate | Narrow, well-evaluated tasks | Exceptions, review burden and domain/language quality |
| Model routing/escalation | Spend more compute where it changes results | Mixed task difficulty | Router/verifier errors, extra calls and latency |
| Quantisation | Reduce numerical precision and memory footprint | Supported model/hardware pairs | Quality, actual kernel speed and memory bottleneck |
| Batching and scheduling | Amortise work and improve useful utilisation | Concurrent compatible requests | Queue delay and tail latency |
| Speculative decoding | Draft candidates and verify with the target model | Suitable generation workloads | Draft overhead, acceptance rate, concurrency |
| Parallelism or phase disaggregation | Match execution stages to resources | Large, sustained workloads | Network transfers, synchronisation and minimum fleet size |
| Capacity changes | Remove idle time/resources or avoid expansion | Variable demand or inefficient provisioning | Cold start, resilience and contractual commitments |
| Model adaptation/distillation | Pay up front for a cheaper recurring task model | High-volume stable tasks | Training/data/evaluation effort and amortisation |
2. Caching requires workload economics, not a switch
Claude documentation specifies matching-prefix reuse, expiry and distinct write/read charges. Current rates depend on the model and platform. vLLM’s prefix caching reuses KV state; its documented benefit targets prefill, not the generation of new output tokens. S10 S14
Author derivation: let w be the write-price multiplier, r the read multiplier, and h the number of subsequent hits on a stable prefix within its valid lifetime. Ignoring other charges:
cached_cost / normal_prefix_cost = w + h × r
uncached_cost / normal_prefix_cost = 1 + h
break_even: h > (w - 1) / (1 - r), where r < 1
Illustration: w=1.25 and r=0.10 pays back with one hit; w=2 and r=0.10 requires two. These are example multipliers, not an assertion that every current model uses them.
Measure cache-read token share, rewrite frequency and cost/task. A request-level hit ratio can hide that only a small fraction of its input was reused. Stable instructions and dynamic task data should be separated where the provider’s cache semantics permit it. Model, tool or configuration changes may invalidate reuse. The September Claude engineering article discusses these operational issues; its optimisation outcomes are provider-authored, not independent proof. S33
3. Routing must account for failures and verification
RouteLLM studies selection between a stronger and a weaker model using preference data. Its published evaluation supports conditional savings; the paper warns about distribution differences and the need for in-domain adaptation. S19
Proposed cascade model:
expected_direct_cost = cheap_attempt_cost
+ verifier_cost
+ escalation_probability × strong_attempt_cost
Add human review, rejected outcomes and latency to the evaluation. A verifier’s false acceptance can be more consequential than a cheap model’s initial error. Start with deterministic task rules where available, then evaluate learned routing or model-based judges against labelled cases.
The October OSCAR preprint is particularly relevant to this decision: it combines iterative attempts with verification in operations-research modelling. Its five benchmark problems do not establish general routing savings for arbitrary enterprise assistants. S20
4. Serving efficiency depends on the phase and workload
Transformer inference has a prompt-processing phase and sequential token generation. Sarathi-Serve’s peer-reviewed work shows how chunking prompt processing can mitigate interference with ongoing generation. Its large improvement factors used 2024 hardware/runtime baselines; do not reuse them as expected 2026 savings. S17
Current vLLM guidance documents a scheduling trade-off: token-batch settings affect prompt latency, generation latency and throughput. Cache pressure can cause recomputation. Use the guidance to form a benchmark hypothesis, not to copy one universal setting. S13
DistServe separates prompt processing and generation across resources and evaluates capacity within latency constraints. The decision to disaggregate must include transfer bandwidth and operating complexity. S18
For non-autoregressive models, Triton supports dynamic batching with an explicitly configured maximum queue delay. That setting creates a measurable throughput-versus-waiting-time trade-off; it is not a free speedup. S16
5. Quantisation and kernel changes need separate checks
vLLM documents multiple quantisation formats and hardware-dependent support. A smaller weight representation does not establish faster execution on a particular workload. S15
Recommended evaluation: separate weight memory, KV-state memory, compute kernel performance and end-to-end service behaviour. Test the intended hardware and request shapes. Include long context, multilingual input, structured extraction and difficult domain cases. For an architecture/model change, evaluate reasoning and multi-step completion as well as short-answer quality.
Kernel optimisation may improve a slow operator while leaving the overall task unchanged because retrieval or tool latency dominates. Translate an operator speedup into its contribution to total execution time before discussing economic benefit.
6. Optimisation is continuous after deployment
Herschel, submitted 30 September 2026, reports Alibaba’s on-demand production profiling across roughly 17,000 traces, more than 120 model variants and over ten accelerator types. Reported inefficiency patterns appeared in 23% of traces. That is not 23% cost savings. Its overhead figures and improvements are author-reported and specific to the evaluated deployment. S21
Our implication: combine inexpensive continuous indicators with bounded diagnostic profiling when needed. Reproduce the triggering workload, test the change, and deploy with review and rollback. Profiling tools are a route to evidence; a generic recommendation alone is not an optimisation result.
7. Benchmark useful capacity, not peak tokens
Proposed protocol:
- Freeze task set, model/version, quality rules and service objectives.
- Replay representative arrival patterns and request lengths; include bursts.
- Sweep concurrency and capacity; report warm/cold and cached/uncached results separately.
- Report accepted tasks, request failures, latency distributions and paid resource hours.
- Compare cost at matched outcomes and service thresholds.
- Validate with an online canary and monitor changed task mix.
Define goodput as the rate of work that meets the stated success and service criteria; distinguish request-level latency goodput from end-to-end business-task goodput. MLPerf Inference v6.1’s September release adds end-to-end RAG and edge agentic tests, reflecting the need to evaluate pipelines and multi-turn workloads. Its scores are comparative system evidence, not corporate ROI. S23
8. Connect the engineering change to a financial action
Proposed explanation pattern: technical mechanism → resource effect → operational/commercial action → financial outcome. For each intervention, state its conditions, measurement and the decision an owner can take.
| Mechanism | Resource effect to measure | Action needed for financial benefit | Outcome to report |
|---|---|---|---|
| Select less context for a product-information answer | Fewer billable input units or less prompt processing | Deploy the evaluated context policy; verify cache and tariff effects | Direct cost per accepted answer, with evidence coverage and review effort |
| Batch or improve serving execution | More accepted tasks per GPU-hour within latency limits | Release capacity, reduce paid hours or defer an otherwise required expansion | Realised spend reduction, avoided expenditure or useful capacity, separately |
| Route simple requests to a cheaper evaluated model | Lower expected inference cost after verification/escalation | Deploy the validated routing policy and maintain its evaluation | Fully loaded cost per accepted task and exception rate |
Hypothetical speaker illustration: an evaluated change lets a workload meet the same quality, demand and latency requirements on one GPU instead of two. If the second equal-priced rented GPU can actually be released for the same paid period, that halves the workload’s GPU rental component. It does not halve total workflow cost, and extra implementation/operating effort must be included. If the fleet remains paid and unchanged, the immediate benefit is capacity, not a lower rental invoice.
A latency reduction alone cannot establish an API saving. Check actual billable units, effective prices and non-token charges. For the workshop’s running example, use an illustrative technical product-information assistant; do not claim a Knauf deployment or disclose unpublished Halbach execution methods.
Presentation implication
Show the intervention table at leadership depth, then explain two mechanisms with one trade-off each. Use a measured Halbach case only after its baseline, scope and release permission are confirmed. Keep routing internals, unpublished architectures and kernel recipes out of the shareable deck.