L21 gave you the $/Mtok identity. This goes inside the denominator. One technical fact — inference happens in two physically different phases — generates the entire economic structure: why output tokens cost 5× input, why you can't have low latency and low cost at once, and why long context is brutally expensive.
In L21 you built serving cost as cost-per-GPU-hour ÷ tokens-per-hour and treated "throughput" as a single slider. That slider hides the most important structure in the whole inference business. Token generation isn't one operation — it's two phases with opposite hardware behavior, and almost every economic fact about LLM serving (the input/output price gap, the latency–cost trade-off, the long-context penalty, the entire frontier of serving optimizations) falls out of that one split. Master it and you can read any inference claim down to the metal.
Core thesis: Prefill (reading your prompt) is compute-bound and parallel — cheap per token. Decode (writing the answer) is memory-bandwidth-bound and sequential — expensive per token. That single asymmetry explains the 5:1 output:input price ratio, makes HBM bandwidth and capacity the binding constraint (not FLOPS), forces a three-way trade-off between latency, throughput, and cost, and makes the KV cache — not the model weights — the thing that actually runs you out of memory at long context. Every name in your thesis is a bet on which side of this split they optimize.
When you send a prompt and get a reply, the GPU does two completely different jobs [Morph]:
Here is the crux, and it's pure L8 roofline: in decode, to generate one token the GPU must stream all ~70–400 billion weights out of HBM. The math is trivial relative to the data moved, so the expensive Tensor cores starve while the memory bus is the bottleneck. Decode speed is set by HBM bandwidth, not FLOPS. This is the single most important sentence in inference hardware — and the reason HBM (L12) is the binding lever, not raw compute.
Now the first economic payoff. Look at any API price sheet — Claude Sonnet: $3/Mtok input, $15/Mtok output (5:1); output is typically priced 3–5× (sometimes up to 8×) the input. [pricing] That ratio isn't a business tactic — it's the prefill/decode split showing up on the invoice:
Investor use: a workload's cost profile depends on its shape, not just its token count. Summarization (huge input, tiny output) is cheap to serve; agentic/reasoning workloads (long chains of generated tokens) are expensive — and "reasoning tokens" are just more decode. When you model a MaaS company's margins, the input:output ratio of its traffic matters as much as its volume. This is why coding and agent products strain unit economics while search-style products don't.
Why doesn't decode re-read the whole prompt for every new token? Because the model caches the intermediate attention state — the keys and values for every token seen so far — in the KV cache. Without it, generating token 1000 would re-process tokens 1–999 every step (O(n²) forever). The KV cache trades memory for compute — and that memory bill is enormous and grows with every token [KV cache]:
Two facts make this the real constraint: the KV cache grows linearly with context length and linearly with the number of concurrent users (batch). At long context it dwarfs the model weights themselves and is what actually caps how many requests a GPU can serve at once — which, via L21, directly caps throughput and sets $/Mtok. Move the sliders to feel it:
The lesson of the slider: push context up and watch the per-user cost explode (a 128k-token request can need tens of GiB by itself); push batch up and you hit the wall fast. Since throughput needs big batches but long context starves them, KV memory is the tightening vise on inference economics — and the reason every frontier optimization below is, at heart, a fight over KV cache. [serving optimization]
Throughput in decode comes from batching — serving many users' tokens in one weight-read, amortizing the memory-bound cost (this is what continuous batching + PagedAttention in vLLM made efficient, lifting batch sizes 2–4× by managing KV like OS memory pages) [vLLM]. But batching fights latency, and the KV cache caps the batch. That creates an iron trade-off:
Diligence reframe: when a vendor quotes a cheap $/Mtok, ask "at what latency and what batch size?" Benchmark throughput at batch-512 is irrelevant to an interactive chat product bound to a tight TPOT SLA. Conversely, a consumer chat app and a nightly batch-summarization job have totally different cost curves on the same hardware. Most published cost claims quietly pick the favorable corner of this triangle.
| Technique | What it does | Which problem it attacks |
|---|---|---|
| Continuous batching + PagedAttention | Schedules per-iteration; manages KV like paged virtual memory (vLLM) | Raises batch/throughput; cuts KV waste 2–4× |
| Quantization (FP8/FP4, KV-cache quant) | Fewer bits per weight & per KV entry | Shrinks both the weight-read (decode speed) and KV size |
| GQA / MQA | Share key/value heads across query heads | Shrinks KV cache 4–8× at the architecture level |
| Speculative decoding | A small draft model proposes tokens; big model verifies many at once | Hides decode's memory latency → more tokens per weight-read |
| Disaggregated prefill/decode | Separate GPU pools for compute-bound prefill vs. bandwidth-bound decode | Lets each phase use ideal hardware; +TTFT & +TPOT together |
| Prefix / KV cache reuse | Share KV for common prompt prefixes across requests | Skips repeated prefill; cuts cost for shared-context traffic |
Disaggregation is the most strategically interesting: because prefill wants FLOPS and decode wants bandwidth, splitting them onto different machines lets you buy the right hardware for each — and production systems like ByteDance's Mooncake report up to ~5× throughput under latency constraints for long-context traffic [Mooncake/MicroServe]. This is why NVIDIA built Dynamo (disaggregated serving) and why cluster design is bifurcating into prefill-optimized and decode-optimized fleets.
Add to THESIS.md: the inference-demand case for every accelerator/memory name now has a mechanism, not just a vibe. The durable question per name is which corner of the latency–throughput–cost triangle, and which side of the prefill/decode split, does this company's hardware win — and is that where the volume is going? Long-context + agentic/reasoning workloads push demand toward decode + KV (favoring HBM capacity); that's the directional bet to track.
Read first: Morph — "LLM Inference: Prefill, Decode, KV Cache & Cost" for the clearest end-to-end mechanics-to-cost walkthrough. Then, the canonical papers: vLLM / PagedAttention (the batching breakthrough) and Sarathi-Serve (the throughput–latency trade-off formalized). Track current serve-cost teardowns via SemiAnalysis (in RESOURCES.md).
Select the best answer for each.
1. The decode phase of inference is bottlenecked primarily by:
2. Output tokens are priced ~5× input tokens fundamentally because:
3. The KV cache is significant for inference economics because it:
4. In the latency–throughput–cost triangle, a larger batch size gives you:
5. Disaggregated prefill/decode serving works because the two phases:
6. Long-context, agentic, and reasoning workloads push hardware demand toward: