Lesson 22 · Chips & LLMs · Inference deep dive

The Inference Engine: Prefill, Decode & the Cost Structure They Generate

L21 gave you the $/Mtok identity. This goes inside the denominator. One technical fact — inference happens in two physically different phases — generates the entire economic structure: why output tokens cost 5× input, why you can't have low latency and low cost at once, and why long context is brutally expensive.

Builds on: L8 (roofline), L12 (HBM), L21 ($/Mtok) Skill: read any serving claim down to physics

In L21 you built serving cost as cost-per-GPU-hour ÷ tokens-per-hour and treated "throughput" as a single slider. That slider hides the most important structure in the whole inference business. Token generation isn't one operation — it's two phases with opposite hardware behavior, and almost every economic fact about LLM serving (the input/output price gap, the latency–cost trade-off, the long-context penalty, the entire frontier of serving optimizations) falls out of that one split. Master it and you can read any inference claim down to the metal.

Core thesis: Prefill (reading your prompt) is compute-bound and parallel — cheap per token. Decode (writing the answer) is memory-bandwidth-bound and sequential — expensive per token. That single asymmetry explains the 5:1 output:input price ratio, makes HBM bandwidth and capacity the binding constraint (not FLOPS), forces a three-way trade-off between latency, throughput, and cost, and makes the KV cache — not the model weights — the thing that actually runs you out of memory at long context. Every name in your thesis is a bet on which side of this split they optimize.

01 — The Two Phases

Prefill vs. Decode: One Engine, Two Physics

When you send a prompt and get a reply, the GPU does two completely different jobs [Morph]:

Phase 1 · Prefill

Reading the prompt

  • Processes all input tokens in one parallel pass → produces the 1st output token.
  • Compute-bound: a big dense matrix-multiply; the GPU's FLOPS are the limit. Attention is O(n²) in prompt length.
  • High arithmetic intensity → rides the compute roof (L8). Hardware is well-utilized.
  • Sets TTFT (time-to-first-token) — the "thinking…" pause.
Phase 2 · Decode

Writing the answer

  • Generates output one token at a time, each depending on the last (autoregressive, sequential).
  • Memory-bandwidth-bound: every token re-reads the entire model's weights from HBM to produce one token.
  • Low arithmetic intensity → stuck on the memory roof (L8). Compute units sit idle waiting for memory.
  • Sets TPOT/ITL (time-per-output-token) — the streaming speed.

Here is the crux, and it's pure L8 roofline: in decode, to generate one token the GPU must stream all ~70–400 billion weights out of HBM. The math is trivial relative to the data moved, so the expensive Tensor cores starve while the memory bus is the bottleneck. Decode speed is set by HBM bandwidth, not FLOPS. This is the single most important sentence in inference hardware — and the reason HBM (L12) is the binding lever, not raw compute.

02 — The Economic Shadow #1: Why Output Costs 5× Input

The Price Asymmetry Is Physics, Not Greed

Now the first economic payoff. Look at any API price sheet — Claude Sonnet: $3/Mtok input, $15/Mtok output (5:1); output is typically priced 3–5× (sometimes up to 8×) the input. [pricing] That ratio isn't a business tactic — it's the prefill/decode split showing up on the invoice:

Investor use: a workload's cost profile depends on its shape, not just its token count. Summarization (huge input, tiny output) is cheap to serve; agentic/reasoning workloads (long chains of generated tokens) are expensive — and "reasoning tokens" are just more decode. When you model a MaaS company's margins, the input:output ratio of its traffic matters as much as its volume. This is why coding and agent products strain unit economics while search-style products don't.

03 — The KV Cache: What Actually Fills Your HBM

The Hidden Memory Hog

Why doesn't decode re-read the whole prompt for every new token? Because the model caches the intermediate attention state — the keys and values for every token seen so far — in the KV cache. Without it, generating token 1000 would re-process tokens 1–999 every step (O(n²) forever). The KV cache trades memory for compute — and that memory bill is enormous and grows with every token [KV cache]:

KV bytes / token = 2 × layers × (kv_heads × head_dim) × precision_bytes ×2 for K and V. The (kv_heads × head_dim) term is why GQA (grouped-query attention) exists: sharing KV heads shrinks this 4–8× vs. full multi-head attention.

Two facts make this the real constraint: the KV cache grows linearly with context length and linearly with the number of concurrent users (batch). At long context it dwarfs the model weights themselves and is what actually caps how many requests a GPU can serve at once — which, via L21, directly caps throughput and sets $/Mtok. Move the sliders to feel it:

KV-Cache Memory-Wall Calculator

How many concurrent users fit on one node before the KV cache overflows HBM? (Architectures use GQA, FP16 KV. Budget = HBM left for KV after weights.)

KV per token
320
KiB · per user, per token
KV per user (at context)
2.5
GiB · grows with context
Total KV at this batch
80
GiB of 640 budget

The lesson of the slider: push context up and watch the per-user cost explode (a 128k-token request can need tens of GiB by itself); push batch up and you hit the wall fast. Since throughput needs big batches but long context starves them, KV memory is the tightening vise on inference economics — and the reason every frontier optimization below is, at heart, a fight over KV cache. [serving optimization]

04 — The Economic Shadow #2: The Latency–Throughput–Cost Triangle

Pick Two. You Cannot Have All Three.

Throughput in decode comes from batching — serving many users' tokens in one weight-read, amortizing the memory-bound cost (this is what continuous batching + PagedAttention in vLLM made efficient, lifting batch sizes 2–4× by managing KV like OS memory pages) [vLLM]. But batching fights latency, and the KV cache caps the batch. That creates an iron trade-off:

The serving trilemma
LOW LATENCY HIGH THROUGHPUT LOW COST small batch = fast but $$$ big batch = cheap but slow batch size dials along this edge
Larger batches amortize the memory-bound weight-read across more users → higher throughput and lower $/token, but each user waits longer (higher latency). To hit a tight latency SLA you must run smaller batches → fewer tokens per GPU-hour → higher $/Mtok. A latency guarantee is literally a cost. [Sarathi-Serve]

Diligence reframe: when a vendor quotes a cheap $/Mtok, ask "at what latency and what batch size?" Benchmark throughput at batch-512 is irrelevant to an interactive chat product bound to a tight TPOT SLA. Conversely, a consumer chat app and a nightly batch-summarization job have totally different cost curves on the same hardware. Most published cost claims quietly pick the favorable corner of this triangle.

05 — The Frontier: How Operators Bend the Curve

Every Optimization Is a Fight Over the Split or the KV Cache

TechniqueWhat it doesWhich problem it attacks
Continuous batching + PagedAttentionSchedules per-iteration; manages KV like paged virtual memory (vLLM)Raises batch/throughput; cuts KV waste 2–4×
Quantization (FP8/FP4, KV-cache quant)Fewer bits per weight & per KV entryShrinks both the weight-read (decode speed) and KV size
GQA / MQAShare key/value heads across query headsShrinks KV cache 4–8× at the architecture level
Speculative decodingA small draft model proposes tokens; big model verifies many at onceHides decode's memory latency → more tokens per weight-read
Disaggregated prefill/decodeSeparate GPU pools for compute-bound prefill vs. bandwidth-bound decodeLets each phase use ideal hardware; +TTFT & +TPOT together
Prefix / KV cache reuseShare KV for common prompt prefixes across requestsSkips repeated prefill; cuts cost for shared-context traffic

Disaggregation is the most strategically interesting: because prefill wants FLOPS and decode wants bandwidth, splitting them onto different machines lets you buy the right hardware for each — and production systems like ByteDance's Mooncake report up to ~5× throughput under latency constraints for long-context traffic [Mooncake/MicroServe]. This is why NVIDIA built Dynamo (disaggregated serving) and why cluster design is bifurcating into prefill-optimized and decode-optimized fleets.

06 — Investment Synthesis

What the Split Means for Each Thesis Name

SK Hynix / HBM — the decode tax collector
Decode is memory-bandwidth-bound and the KV cache is memory-capacity-bound, so both axes of inference scaling pull on HBM. As context windows and concurrency grow, HBM content per accelerator rises structurally — the single cleanest read-through from inference mechanics to a thesis name (L12). Bull intact while HBM stays the binding constraint.
AMD — the capacity wedge, explained
MI300X/MI350X lead on HBM capacity, which directly relaxes the KV-cache wall: fit longer context or larger batch in fewer GPUs → lower $/Mtok on memory-bound serving. This lesson is the mechanism behind the L16 wedge — and why AMD competes best precisely where decode/KV dominate, not in compute-bound training.
NVIDIA — moat is the whole engine
NVDA's edge isn't just FLOPS; it's the software (vLLM/TensorRT-LLM ecosystem), NVLink bandwidth for multi-GPU decode of huge models, and now Dynamo disaggregation — i.e. owning the throughput lever and the serving stack. The risk remains: these are engineering advantages competitors and open-source (vLLM, SGLang) chip at, not laws of physics.
Broadcom / ASICs — built for one corner
A custom inference ASIC can be tuned to the exact prefill/decode ratio and KV pattern of a stable, high-volume workload (e.g. Google's TPU for its own serving) — stripping out GPU generality to win $/Mtok. The build-vs-buy line from L21 is really "does our traffic sit still enough in this triangle to justify fixed silicon?"

Add to THESIS.md: the inference-demand case for every accelerator/memory name now has a mechanism, not just a vibe. The durable question per name is which corner of the latency–throughput–cost triangle, and which side of the prefill/decode split, does this company's hardware win — and is that where the volume is going? Long-context + agentic/reasoning workloads push demand toward decode + KV (favoring HBM capacity); that's the directional bet to track.

Primary Source

Go Deeper

Read first: Morph — "LLM Inference: Prefill, Decode, KV Cache & Cost" for the clearest end-to-end mechanics-to-cost walkthrough. Then, the canonical papers: vLLM / PagedAttention (the batching breakthrough) and Sarathi-Serve (the throughput–latency trade-off formalized). Track current serve-cost teardowns via SemiAnalysis (in RESOURCES.md).

Comprehension Check

Quiz — 6 Questions

Select the best answer for each.

1. The decode phase of inference is bottlenecked primarily by:

Raw FLOPS, since each token needs huge parallel matrix math
HBM bandwidth, since each token re-streams the model weights
Network latency between the user device and the data center
CPU clock speed, which orchestrates the whole generation loop

2. Output tokens are priced ~5× input tokens fundamentally because:

Providers mark up output to capture extra margin on demand
Output text is longer on average than the input prompt sent
Input is parallel prefill while output is sequential decode
Output tokens are stored permanently and input tokens are not

3. The KV cache is significant for inference economics because it:

Permanently stores every user conversation for later retraining
Grows with context and batch, capping concurrent users per GPU
Replaces the model weights entirely once generation has begun
Shrinks steadily as more output tokens are generated over time

4. In the latency–throughput–cost triangle, a larger batch size gives you:

Higher throughput and lower cost, but higher per-user latency
Lower latency and lower cost, but reduced total throughput
All three improvements at once with no trade-off whatsoever
Higher latency and higher cost, with throughput held constant

5. Disaggregated prefill/decode serving works because the two phases:

Use identical hardware, so splitting them simplifies scheduling
Have opposite needs — compute-bound versus bandwidth-bound
Must run on separate continents for data-sovereignty reasons
Cannot share a KV cache, so they require physical isolation

6. Long-context, agentic, and reasoning workloads push hardware demand toward:

Compute-bound prefill, favoring pure FLOPS over memory capacity
Decode and KV cache, favoring HBM bandwidth and capacity
CPU-only inference, since generation is mostly simple bookkeeping
Smaller models exclusively, removing the need for advanced memory
From your instructor: The one idea to keep — inference is two phases: prefill (compute-bound, parallel, cheap, = input tokens) and decode (memory-bound, sequential, expensive, = output tokens), and the KV cache is what fills your HBM. From that single split you can re-derive the 5:1 price ratio, the latency–throughput–cost triangle, the long-context penalty, and why HBM — not FLOPS — is the binding lever. Ask me anything: how speculative decoding actually verifies tokens, how GQA shrinks the KV cache, or how to estimate a specific model's serving cost at a target latency.