The Inference Economics Stack
Day 83 was the DEMAND side of Agent FinOps -- your routing, caching and budget levers. Day 84 was the people who run the fleet. Today is the SUPPLY side: the silicon and serving layer that actually sets the cost-per-token your FinOps levers push against. As frontier models compress to a near-tie (within ~3%), you can no longer out-model a rival -- so the margin war moves down the stack to who can serve a token cheapest. And the numbers moved hard this month: AWS shipped the 192-core Graviton5 on June 15, Amazon's custom-silicon run-rate crossed $20B, and SpaceX's S-1 revealed Anthropic is paying xAI $1.25B a month to rent 300MW of compute. This issue walks the three layers that set cost-per-token -- the custom silicon, the serving engine, and the optimisation techniques (quantisation, speculative decoding, disaggregated prefill) -- and why the GPT-4-class token just fell ~1,000x in three years.
llama.cpp -- the engine that drives cost-per-token to ~$0
On a day about who serves a token cheapest, the trending project is the one that drives the price to the floor. llama.cpp (Georgi Gerganov / ggml-org) crossed 100,000 GitHub stars in March 2026 -- the fastest open-source AI project ever to reach the mark, quicker than PyTorch (~7 years) or TensorFlow (~8 years). It is the C/C++ inference engine that quietly powers most of the local-AI world (Ollama, LM Studio and dozens of apps all sit on top of it), and its momentum is staggering: 700+ contributors merged 3,800+ pull requests in 2025, roughly 3x the PR throughput of NVIDIA's fully-funded TensorRT-LLM. It runs the same GGUF model across Apple Metal, NVIDIA CUDA, AMD ROCm, Intel SYCL, Vulkan and ARM NEON -- from a Raspberry Pi 5 to an 8x-GPU server -- and now ships WebGPU inference in the browser, with Hugging Face Inference Endpoints supporting GGUF out of the box. Why it matters for this issue: local inference on llama.cpp costs roughly $0.002 per million tokens in electricity, versus $2.50-15.00/M through cloud APIs. That is the cost-per-token war reaching the desktop -- the cleanest proof that the supply side, not the model, is where the economics are now won. (OpenClaw still tops the raw OSS star charts at 210K+ as the local-first foil -- viral, but a personal agent, not the serving engine underneath it.)
- 100K stars — fastest OSS AI project ever to 100K (March 2026) -- beat PyTorch & TensorFlow
- ~$0.002 / M — local inference in electricity vs $2.50-15.00/M cloud API -- the floor
- RPi 5 -> 8x GPU — one GGUF model across Metal / CUDA / ROCm / SYCL / Vulkan / ARM + WebGPU
1. Why the supply side is suddenly the battleground
Day 83 gave you the demand-side levers -- model routing, caching, budget caps, tagging. Every one of those levers pushes against a floor that something else sets: the cost-per-token of actually running the model. That floor is the supply side, and it is built from three stacked layers -- the silicon the model runs on, the serving engine that schedules the work, and the optimisation techniques (quantisation, speculative decoding) that squeeze more tokens out of each chip-second. Your FinOps savings are bounded by how cheap this stack can make a token; the cheaper the floor, the more headroom your routing and caching actually buy you.
Two things make this the battleground of mid-2026. First, model performance has compressed to a near-tie -- the top models sit within roughly 3% of each other -- so you can't win on capability; you win on unit economics. Second, the spend has shifted decisively: industry analysts now put 55-80% of enterprise AI GPU budget on INFERENCE, not training. Training is a one-time capital event; serving is a bill that accrues every hour, every day, for the life of the product. Whoever owns the cheapest path from prompt to token owns the margin -- which is exactly why Amazon, Google, NVIDIA, AMD and a wave of start-ups are pouring tens of billions into custom silicon aimed squarely at the decode phase.
The headline proof that this is now a strategic, balance-sheet-level fight: SpaceX's S-1 filing disclosed that Anthropic is paying xAI $1.25B a month to rent 300MW of compute (220,000 NVIDIA GPUs at the Colossus 1 site near Memphis) through May 2029 -- over $40B from one frontier lab to a direct rival, capped at 11% of the park and terminable on 90 days' notice. When labs rent compute from competitors at that scale, the message is clear: the binding constraint on agentic AI is no longer the model -- it is the cost and availability of the inference substrate underneath it.
2. The silicon layer -- custom chips break the GPU monopoly
For three years inference meant NVIDIA GPUs. In 2026 the hyperscalers' custom silicon turned that into a real contest -- because designing a chip for the decode phase (memory-bound, latency-sensitive) rather than for training (compute-bound) changes the price-performance maths. AWS shipped the 192-core Graviton5 on June 15 with a 5x larger cache, ~35% faster ML inference and M9g instances up to 25% faster than the prior generation, explicitly pitched for 'the agentic AI era.' Its Trainium line is the bigger story: Trainium2 runs ~30% better price-performance than comparable GPUs and is largely sold out, Trainium3 (shipping since early 2026) is another 30-40% better and nearly fully subscribed, and Amazon's custom-silicon run-rate has crossed $20B (Andy Jassy says it would be ~$50B if sold externally). Anthropic has committed over $100B to AWS over ten years and up to 5GW of Trainium capacity.
Google reframed the whole category around inference. Ironwood -- its 7th-gen TPU -- went GA as 'the first TPU for the age of inference,' and at Cloud Next 2026 Google previewed an eighth-generation split into a training chip (TPU 8t) and a dedicated inference chip (TPU 8i) on TSMC's 2nm process, claiming up to 2.8x training gains and an 80% inference price-performance improvement over Ironwood. The bifurcation is the tell: general-purpose silicon is giving way to chips purpose-built for serving.
The incumbents are not standing still, but the timing favours challengers near-term. NVIDIA's Rubin generation slipped roughly a quarter (ramp now around Q3 2026), AMD's MI400 -- 432GB of HBM4 and 19.6 TB/s of bandwidth -- goes to hyperscalers first with broad availability in 2027, and specialised entrants like Groq are attacking the decode phase directly (its LPU claims ~35x more inference throughput per megawatt than HBM-based GPUs). For an agent fleet, the practical takeaway is that 'which chip' is now a cost lever you can actually pull: the same Claude or open-weight model served on Trainium, a TPU 8i or a Groq LPU can carry a materially different cost-per-token than the default NVIDIA path.
3. The serving engine -- software that 5-8x's your throughput
Silicon sets the ceiling; the serving engine decides how much of it you actually use. The three that matter in 2026 -- vLLM, SGLang and TensorRT-LLM -- serve the same purpose but make different architectural bets, and the rule of thumb is simple: vLLM for the fastest path to production, TensorRT-LLM for peak NVIDIA throughput, SGLang for heavy multi-turn prefix reuse. All three now ship continuous (in-flight) batching, where a new request joins the running batch the moment a sequence slot frees up instead of waiting for the whole batch to drain -- the single biggest GPU-utilisation win of the last two years.
Three more primitives separate a cheap deployment from an expensive one. PagedAttention / KV-cache paging treats the attention cache like virtual memory so you stop wasting VRAM on reserved-but-unused slots. Prefix caching reuses the KV cache for shared prompt prefixes -- decisive for agents, whose long system prompts and tool schemas repeat on every call (this is the serving-side mirror of the caching lever from Day 83). And disaggregated prefill splits the compute-bound prefill phase and the memory-bound decode phase onto separate devices so they stop interfering -- now standard in vLLM and SGLang, and the reason large deployments get stable decode latency under load.
For an agentic workload the serving engine is not a back-office detail -- it is a first-class FinOps decision. Long, repetitive system prompts make prefix caching worth more to an agent than to a chatbot; bursty, parallel sub-agent fan-out makes continuous batching and disaggregated prefill the difference between 40% and 90% GPU occupancy. Picking the engine and turning on these features is often a larger cost win than swapping the model -- and it is entirely under your control.
4. The optimisation layer -- where the $0.40 token comes from
The last layer is the math trick that collapsed the price: serve the model in lower precision and predict more tokens per pass. Quantisation is the workhorse. Moving from FP16 to FP8 roughly halves the cost of an 8x-H100 node -- from about $1.90 to $0.95-1.10 per million tokens -- with negligible quality loss for most production traffic. Going further, 4-bit (AWQ) quantisation cuts VRAM by roughly 75%, which doesn't just save memory -- it lets the same model fit on a cheaper GPU class entirely, changing the hardware line item.
Speculative decoding attacks throughput from the other side: a small, cheap 'draft' model proposes several tokens, and the large target model verifies them in parallel, accepting the ones it agrees with. For output-heavy work like code generation and long-form writing that is a 2-4x speed-up at no quality cost. Stack the techniques together -- FP8 + FlashAttention 3 + continuous batching + speculative decoding on an H100 -- and you get 5-8x better cost-efficiency than naive FP16 with static batching. That multiplier, compounded across the whole industry, is what took a GPT-4-class token from ~$30/M in early 2023 to ~$0.40/M today: a roughly 1,000x collapse, one of the fastest in computing history.
The discipline that ties it back to Day 83: cheaper tokens are not the goal -- cheaper SUCCESSFUL OUTPUTS are. The FinOps Foundation's warning is that instrumenting cost-per-token without a quality gate is a trap; a 4-bit model that fails 5% more tasks can cost more per completed job than the FP8 version it replaced. So the supply-side stack (silicon -> serving engine -> quantisation/speculative decoding) is what sets your floor, but you still measure the win in cost-per-successful-task on the same OTEL->WORM telemetry stream you built for the regulator (Day 81), the buyer (Day 82) and your FinOps dashboard (Day 83).
With frontier models within ~3% of each other (Fable 5 / Opus 4.8 / GPT-5.6 / Gemini 3.5 Flash), capability is no longer the moat -- and Day 83 showed the demand-side levers are now table stakes. The next margin war is fought on the SUPPLY side: who can serve a token cheapest. That is why the money is moving into the inference substrate -- Amazon's custom-silicon run-rate past $20B with >$225B in Trainium commitments, Google splitting its TPU line into dedicated training (8t) and inference (8i) chips, AMD's MI400 and Groq's LPU attacking the decode phase, and Anthropic renting 300MW from xAI at $1.25B/month because the binding constraint is compute, not models. For a builder, the strategic read is that 'which chip + which serving engine + which quantisation' is now a first-class product decision, not an infra afterthought -- and the open-source serving stack (vLLM/SGLang) plus llama.cpp means the cheapest-token frontier is available to everyone, not just the hyperscalers. As model performance plateaus, the durable advantage is the lowest cost per successful, audited token -- the unit-economics race the IPO market (SpaceX priced, Anthropic and OpenAI both in the S-1 pipeline) is now pricing directly.
Pick the engine for your workload -- vLLM for fastest-to-prod, TensorRT-LLM for peak NVIDIA throughput, SGLang for prefix-heavy agents -- then turn on continuous batching, prefix caching and disaggregated prefill. For repetitive agent prompts that is often a bigger cost win than swapping the model, and it is entirely under your control.
FP8 roughly halves cost-per-token at negligible quality loss; 4-bit can drop you to a cheaper GPU class; speculative decoding adds 2-4x on output-heavy work. But validate on your golden set -- a lower-precision model that fails more tasks can cost MORE per completed job. The metric is cost-per-successful-output, not cost-per-token (Day 83).
The same model on Trainium3, a TPU 8i, a Groq LPU or llama.cpp on your own hardware can carry a very different cost-per-token than the default NVIDIA path. Benchmark at least one non-NVIDIA option per workload, and for low-volume or privacy-bound tasks, remember local inference runs at ~$0.002/M in electricity versus $2.50-15/M in the cloud.