Select the operating model from demand, constraints and lifecycle cost
Recommendation: Compare managed APIs, dedicated managed capacity and self-operated infrastructure using the same task-quality and service requirements. Prefer reversible choices while demand is uncertain; consider committed or self-operated capacity when evidence supports its utilisation and operational burden.
1. Compare operating models on equal terms
| Model | Economic structure | Attractive conditions | Hidden work or risk |
|---|---|---|---|
| Consumption API | Usage-linked charges; platform operations included in price | Uncertain/bursty demand, fast experimentation | Provider dependency, rate/tier changes, limited low-level optimisation |
| Dedicated managed endpoint | Capacity/time charges plus managed operations | Stable demand with isolation or performance needs | Idle capacity and minimum commitments |
| Self-operated rented GPUs | Resource-time charges plus your operating effort | Sustained demand, controllable models, relevant engineering capacity | Scheduling, observability, updates, resilience and incident ownership |
| Owned/on-premises infrastructure | Capital, lifecycle operations and energy | Explicit locality/control requirements and credible utilisation plan | Acquisition lead time, depreciation, maintenance and reserve capacity |
| Hybrid | Combination of fixed and variable costs | Stable base load with variable peaks or differentiated tasks | Routing, evaluation, reconciliation and duplicate integration work |
This is an analytical comparison, not a vendor ranking. Open weights do not automatically imply unrestricted commercial licensing. Check the specific model and dependencies before deployment.
2. Calculate break-even at a matched service level
Evidence boundary: this chapter supplies an analytical method and hypothetical arithmetic. It does not contain a measured provider ranking, approved GDPR configuration matrix or fully populated owned-hardware TCO. See 09_provider_eligibility_tokens_and_tco.md [local reference omitted] for the full method and missing inputs.
Author-created simplified model:
API monthly cost = Q × a
self-operated monthly cost = F + Q × v
break-even volume Q* = F / (a - v), if a > v
Q is accepted tasks at matched quality; a is effective API cost per accepted task; F includes fixed capacity and explicitly scoped operating cost; v is incremental self-operated cost per accepted task. Include unsuccessful attempts in the relevant cost numerator.
Illustration: a=€0.05, v=€0.008 and F=€4,000/month produce a break-even near 95,238 accepted tasks/month. At 100,000 tasks, the simplified costs are €5,000 versus €4,800. The €200 difference is fragile; modest migration or operating effort can reverse it. These inputs are hypothetical.
Real capacity costs are stepped, not perfectly linear. A second replica, a larger model, peak demand or redundancy can change F abruptly. If low traffic allows genuine scale-to-zero, F itself changes with demand. Use measured capacity curves rather than treating this equation as a purchasing decision.
3. Price useful output from rented capacity
compute_cost_per_successful_task
= paid_GPU_and_supporting_resource_cost / successful_tasks_delivered
Or, at steady matched conditions:
compute_cost_per_task ≈ resource_cost_per_hour / successful_tasks_per_hour
The hourly rate must cover every active resource used in the numerator. The task rate must respect agreed latency, quality and failure criteria. Account for idle hours, warm-up, maintenance, spare replicas and the resources used by retrieval or other models.
DistServe and Sarathi-Serve establish why service-constrained capacity is a more useful comparison than unconstrained peak throughput. Their historical benchmark factors cannot price a current rental fleet. S18 S17
For Halbach’s rented-GPU model, an efficiency gain has three distinct possible consequences: reduce rented resources, accommodate more valuable work, or avoid the next capacity increase. Identify which one a customer can actually realise.
4. Treat deployment features and discounts as constrained options
AWS’s current pricing page describes discounted batch inference for selected models. Its batch documentation describes asynchronous independent records and states that tool calling and structured-output response format are unsupported on that path. Its prompt-cache documentation states that caching is not supported with the batch inference API. Do not stack independently advertised discounts without checking compatibility. S12 S11 S32
Similarly, assess:
- Availability of the exact model in the required region and service tier.
- Whether a contract’s minimum commitment is consumed by the intended traffic.
- Cache lifetime, minimum eligible context and billing categories.
- Support, rate limits, redundancy and deployment-update responsibilities.
- Exportability of logs, evaluations, model artefacts and application interfaces.
Recommendation: Calculate expected savings over actual demand scenarios and the contract period. Do not buy a commitment to improve a unit rate if it increases expected total expenditure.
5. Map the market by job performed
| Category | Examples inspected | Useful role | Boundary |
|---|---|---|---|
| Billing and FinOps standards | FinOps Framework; FOCUS | Normalise cost records and financial processes | Cannot define business success automatically |
| AI gateways | LiteLLM | Credential, routing and budget controls | Depend on traffic coverage and configured enforcement |
| LLM observability/evaluation | Langfuse; OpenTelemetry conventions | Diagnose calls and connect usage with evaluations | Estimated cost needs reconciliation |
| Serving runtimes | vLLM; NVIDIA Triton | Execute supported models efficiently | Require model/hardware/workload-specific configuration |
| Managed inference services | Amazon Bedrock; Claude API | Purchase hosted model capabilities | Service-specific rates and feature constraints |
| Specialist engineering | Model, workflow and runtime optimisation | Change the economic behaviour of a workload | Results require access, benchmarks and deployable changes |
Examples illustrate category boundaries; they are not an exhaustive procurement shortlist or endorsement. Re-check feature availability and paid versus open-source editions for a real implementation. Sources S04 S05 S07 S08 S13 S16 S12 S10.
6. What is current as of October
The Bedrock pricing page retrieved for this research contains an effective-date note for October 7 reducing Sonnet 5.5 cache-read pricing by 50% relative to its prior rate. The note does not print a year or establish savings for the whole request. It is a live-page observation, not a separately dated release announcement. S12
Author derivation: if that affected cache-read component represents x% of a workload’s cost, halving only that component reduces the total by 0.5x%, assuming other rates and usage remain fixed. A 20% component share would imply a 10% total reduction, not 50%.
The market lesson is stronger than a single price point: cached/uncached mix, workload behaviour and platform restrictions can matter as much as the headline model rate. Refresh the actual rate schedule before the workshop.
7. Establish eligibility before comparing full lifecycle cost
GDPR and DSGVO refer to the same regulation. Review the workload, AVV/DPA, retention/training use, subprocessors, security and international transfers for the exact service configuration. EDPB guidance describes transfer conditions in addition to ordinary GDPR obligations. S34 Regional storage and regional inference require separate verification; OpenAI’s documentation illustrates that distinction and relevant exclusions. S35
Compare approved configurations on the same representative tasks and quality/latency requirements. For owned hardware, include acquisition/residual value or depreciation in the appropriate financial view, energy/cooling, supporting resources, resilience, maintenance and operating labour. For rented GPUs, use paid resource charges without adding embedded electricity again. Include integration, supporting workflow services and human review on all sides.
The decisive denominator is accepted tasks delivered over the cost period, with failed attempts included in expenditure. A hardware quote, peak tokens/second or API rate card alone cannot establish the result. The detailed cost matrix, cash-versus-depreciation distinction, token fields and input checklist are in chapter 09.
8. Match the optimisation lever to what the customer controls
Analytical map: capabilities depend on the exact service. More execution control also creates more operating responsibilities.
| Deployment | Typical customer-controlled levers | Boundary and decision |
|---|---|---|
| Managed consumption API | Calls/steps, context, supported caching, model/routing, tier/batch choices | Provider controls most execution internals. Optimise measured billed consumption and complete workflow cost. |
| Managed dedicated endpoint | Application levers plus exposed model, capacity and scaling settings | Runtime/kernel control varies. Verify minimum capacity, service constraints and what can actually be changed. |
| Self-operated rented or owned inference | Application levers plus supported precision, batching, scheduling, kernels and capacity configuration | Customer takes responsibility for deployment, evaluation, resilience and operations. Compare realised useful capacity and full cost. |
Use this map to explain when specialist inference engineering is relevant. Establish who has authority to deploy a change, remove resources or change a commitment. A nominal technical gain is not a savings claim until its operational and commercial consequences are identified.
9. Engineering spend and delivery measurement is an adjacent market category
Navigara describes repository-derived output measurement, usage attribution, budget controls and optional routing; Faros also describes engineering token attribution and routing on its own site. S50 S49 These are public product propositions, not a verified capability ranking. Routing therefore overlaps the existing model-selection landscape, while engineering-management analytics address a broader buyer question.
For the workshop, name the category rather than presenting a vendor catalogue. Distinguish financial visibility, workflow improvement and inference execution optimisation. Public model-routing claims alone do not establish Halbach’s differentiation. Use matched-quality task evidence, operating effort and realised financial consequences; keep proprietary implementation outside the public deck. Chapter 10 [local reference omitted] contains the detailed assessment.