Lesson 08 · Chips & LLMs

LLM Inference at Scale

PagedAttention, continuous batching, speculative decoding, and quantization — the systems layer that determines actual GPU economics

⏱ ~50 min 📍 vLLM · PagedAttention · Continuous batching · Speculative decoding · Quantization ← Lesson 07 → Lesson 09

From architecture to economics

Lesson 7 established that inference is memory-bandwidth-bound and that the KV cache is the dominant memory consumer. This lesson explains the systems layer that sits between the Transformer architecture and the hardware: the inference serving frameworks. The insight: a naive implementation of the H100 for LLM inference achieves 20–30% GPU utilization. The techniques in this lesson push that to 80–90%. The difference is 3–4× more tokens per GPU per dollar — the most important cost lever in AI deployment.

Part 1 — Prefill vs. Decode: Two Completely Different Workloads

Every inference request has two phases with opposite hardware characteristics

Prefill Phase

Process the prompt — all input tokens simultaneously. Attention is computed over the full prompt in parallel (like training). Builds the initial KV cache.

Arithmetic intensity: HIGH

Batch all prompt tokens together → large matrix multiply → Tensor Cores are busy. Compute-bound at most prompt lengths ≥ 512 tokens.

Optimized for: throughput (tokens/sec). Multiple prompts can be prefilled together.

TTFT (Time To First Token) = prefill time. User perceives this as "thinking."

Decode Phase

Generate output tokens one at a time, autoregressively. Each step loads all model weights + growing KV cache from HBM, computes one token.

Arithmetic intensity: LOW (0.5–50 ops/byte)

Memory-bandwidth-bound (Lesson 7). GPU is waiting for HBM reads, Tensor Cores idle ~99% of the time at batch=1.

Optimized for: latency (tokens/sec/user). Cannot parallelize across tokens — strictly sequential.

ITL (Inter-Token Latency) = decode time per token. User perceives this as generation speed.

Why prefill and decode want to be separated ("disaggregated serving")

Prefill is compute-bound; decode is memory-bandwidth-bound. Running them on the same GPU wastes resources: during decode, the expensive Tensor Cores sit idle. During prefill, memory bandwidth is unused. The next-generation serving architecture (Mooncake by ByteDance, Splitwise by Microsoft Research) disaggregates prefill onto high-TFLOPS GPUs and decode onto high-bandwidth GPUs (or ASICs like Groq's LPU). This is not yet mainstream in 2025 but is the direction high-scale API providers are moving.

Part 2 — Continuous Batching: From 30% to 80%+ GPU Utilization

The biggest single systems improvement in LLM serving history

Static (naïve) batching: group requests into a batch, run until every sequence in the batch finishes generating, then accept new requests. The problem: sequences finish at different times. A short request (50 tokens) finishes long before a long request (2,000 tokens). The GPU sits idle waiting for the long request while it could be serving new users.

Static batching vs. continuous batching — GPU slot utilization over time

t=1

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

t=11

t=12

STATIC BATCHING (naïve)

Slot 1 (Req A)

A · 4 steps

Slot 2 (Req B)

B · 6 steps

Slot 3 (Req C)

C · 12 steps (long response)

Slot 4 (Req D)

IDLE — waiting for batch to complete

Batch 1 ends at t=12 (when C finishes). Req D cannot start until t=13. GPU utilization: ~50% (slots 1,2,4 idle while C runs).

CONTINUOUS BATCHING (vLLM / Orca)

Slot 1 (Req A)

A · 4 steps

E · 3st

F · 3st

G·2

Slot 2 (Req B)

B · 6 steps

H · 4 steps

I·2

Slot 3 (Req C)

C · 12 steps

Slot 4 (Req D)

D·2

J · 6 steps

K · 4 st

New requests added immediately when slots free. GPU slots are nearly always busy. Utilization: ~85–90%.

Continuous batching (from the "Orca" paper, 2022 — adopted as the foundation of vLLM) treats each decode step as an opportunity to replace finished sequences with new ones. This requires no wasted padding between requests. In practice, continuous batching achieves 5–23× better throughput than static batching on serving Llama-class models at realistic production traffic patterns.

Part 3 — PagedAttention: Virtual Memory for the KV Cache

The key insight that made vLLM the dominant inference framework

The KV cache problem: when you allocate memory for a request, you don't know how long it will be. A request could generate 50 tokens or 5,000 tokens. Naïve allocation (pre-allocate max sequence length) wastes enormous memory. Dynamic allocation (grow as needed) fragments memory — the free blocks scattered between allocated blocks become too small to use. In the worst case, 60–80% of HBM was wasted on fragmentation and over-allocation.

Memory layout: naïve contiguous allocation vs. PagedAttention

NAÏVE: Pre-allocate max length — memory fragmentation

—

Red = pre-allocated but unused (wasted). Fragmented free blocks too small for new long requests. Memory waste: 40–60%.

PAGEDATTENTION: Fixed-size blocks, non-contiguous — like OS virtual memory

Blocks allocated one at a time as sequences grow. A page table maps logical KV positions to physical blocks. Memory waste: <4%.

PagedAttention (Kwon et al., UC Berkeley, 2023 — basis of vLLM) borrows the OS virtual memory abstraction. Fixed-size KV "pages" (typically 16 tokens of KV data each) are allocated from a free pool as sequences grow. A block table maps logical position → physical block. During attention computation, the attention kernel gathers blocks from non-contiguous addresses — a minor CUDA kernel modification that enables near-zero memory waste. Combined with continuous batching, vLLM achieves 2–4× better throughput than HuggingFace's naive serving on the same hardware.

Prefix caching — the next PagedAttention extension

Many requests share a common prefix: a system prompt. "You are a helpful assistant. The following is a conversation..." — same 500 tokens, computed fresh for every user. PagedAttention's block structure enables prefix caching: shared KV blocks are computed once and stored persistently in the block pool. New requests that share the same system prompt reuse those pre-computed blocks, skipping their prefill step entirely. In production API serving where every request has the same system prompt, prefix caching reduces prefill compute by 30–70% and TTFT by a similar margin.

Part 4 — Speculative Decoding: Generating Multiple Tokens in One Pass

The insight: verifying N tokens in parallel is not more expensive than verifying 1

Decode is serial: generate token 1, then token 2, then token 3. You cannot generate token 3 until you know token 2. This seems like a hard constraint — but speculative decoding breaks it with a trick.

A small, fast "draft" model generates N candidate tokens quickly (cheap: small model). The large "target" model then verifies all N tokens in one forward pass (because the target model, like the decoder during training, can process all N tokens in parallel with causal masking). If the target model agrees with the draft model on all N tokens, you get N tokens for the cost of ~1. If it disagrees at position k, you keep tokens 1…k–1, resample token k from the target, and restart drafting.

Speculative decoding vs. naïve autoregressive — time slots

NAÏVE AUTOREGRESSIVE — one token per full forward pass

Target model (8×H100)

tok 1

tok 2

tok 3

tok 4

tok 5

5 tokens = 5 full forward passes. 5 × 50ms = 250ms.

SPECULATIVE DECODING (draft γ=4) — 4 candidate tokens per draft cycle

Draft model (1×H100)

Target model (8×H100)

waiting for draft (fast)

✓ 1

✓ 2

✓ 3

✗ 4

3 tokens accepted (tok 1–3) in one target forward pass. Tok 4 rejected: target resamples it, restarts draft from tok 4 onward. Net: 3 tokens per target pass. At 75% acceptance rate: ~3× speedup vs. naïve (2–3× typical in practice).

The mathematical guarantee: speculative decoding produces exactly the same output distribution as the target model alone — it is not an approximation. The proof: rejected tokens are resampled from the target model's true distribution. Accepted tokens were already correct by definition. This is why speculative decoding can be used in production without quality degradation. Google's "Medusa" and "EAGLE" variants train additional draft heads directly on the target model's activations, achieving even higher acceptance rates. NVIDIA's TensorRT-LLM includes speculative decoding as a first-class feature.

Part 5 — Quantization: Trading Precision for Memory and Speed

How to fit a 405B model on 4 H100s instead of 8

Neural network weights are floating-point numbers. FP32 = 4 bytes/weight, BF16 = 2 bytes/weight. But most weights encode information redundantly — you can represent them with fewer bits with minimal quality loss. This is quantization: map the continuous weight distribution to a smaller set of discrete values.

Precision ladder for Llama 3 405B (405 billion parameters)

FP32

Full precision training (reference)

1,620 GB

Rarely used in prod; too large

BF16

Training + standard inference baseline

810 GB

10 × H100 80GB. 0% quality loss

INT8 / FP8

H100 Tensor Core native

405 GB

5 × H100. <0.5% quality loss

INT4 (GPTQ/AWQ)

Edge of practical quality

202 GB

3 × H100. ~1–3% quality loss

INT4 (GGUF/llama.cpp)

CPU/Apple Silicon inference

~200 GB

No GPU needed. Community/edge use.

INT2 (research)

Quality cliff

~100 GB

Significant quality degradation. Not production-ready.

Weight-only vs. weight+activation quantization

Weight-only quantization (W8A16, W4A16): weights stored at low precision, dequantized to BF16 before matrix multiply. Memory savings (smaller model fits in less HBM), but compute still runs in BF16. Used by GPTQ, AWQ, llama.cpp. Simple, reliable, minimal quality loss.

Weight + activation quantization (W8A8, W4A8, FP8): both weights and activations quantized — the actual matrix multiply runs in INT8 or FP8. Requires calibration data to compute per-tensor scaling factors. Gives compute throughput improvements in addition to memory savings (INT8 Tensor Cores on H100: 7,916 TOPS vs. 3,958 TFLOPS BF16 — 2× more ops). Used by TensorRT-LLM's FP8 quantization for production NVIDIA deployments. State-of-art for production inference in 2024–2025.

The 2024 result: Llama 3 405B in FP8 via TensorRT-LLM on 8×H100 SXM5 achieves ~1,300 tokens/sec at batch=32, vs. ~800 tokens/sec in BF16 on the same hardware. FP8 is not an approximation of BF16 — it's a full 2× compute speedup because H100's Tensor Cores do 2× more FP8 ops per second than BF16 ops per second.

Part 6 — The Inference Framework Ecosystem

vLLM, TensorRT-LLM, TGI, llama.cpp — when to use which

Framework	Developer	Best for	Key advantage	Key limitation
vLLM	UC Berkeley / vLLM Project (open-source)	Cloud API serving, high-throughput, diverse models	PagedAttention + continuous batching. Widest model support. OpenAI-compatible API. Python-first, easy to deploy.	Not the fastest peak throughput — TensorRT-LLM is faster for NVIDIA with tuned kernels.
TensorRT-LLM	NVIDIA (open-source)	Maximum NVIDIA throughput, production deployment	Fused CUDA kernels, FP8 support, speculative decoding, in-flight batching. 2–3× faster than vLLM on NVIDIA. Used by all major cloud providers serving on H100.	NVIDIA-only. Requires model compilation (not plug-and-play). Less flexible for new model architectures.
HuggingFace TGI (Text Generation Inference)	HuggingFace (open-source)	HuggingFace ecosystem integration, developer experience	First-class HuggingFace Hub integration. Easy to spin up. Good for prototyping.	Lower throughput than vLLM / TRT-LLM at scale. Less production-optimized.
llama.cpp	Georgi Gerganov (open-source)	CPU / Apple Silicon / edge inference, local deployment	Runs 7B–70B models on MacBook / gaming PC via GGUF quantization. No GPU needed. Active community.	CPU inference: 10–50× slower than GPU. Not scalable for API serving at volume.
SGLang	Stanford / LM-Sys (open-source)	Structured generation, multi-turn, agent workflows	RadixAttention for prefix caching across complex request patterns. Fast for LLM programs (constrained decoding, function calling).	Newer ecosystem, fewer deployment examples than vLLM.

Part 7 — Inference Cost Trajectory and Investment Implications

100× cost reduction in 18 months — and what comes next

GPT-4 class 1M output tokens cost — market price trajectory

Mar 2023

GPT-4 launch — $60/1M tokens output

$60.00

Nov 2023

GPT-4 Turbo — $30/1M tokens

$30.00

Apr 2024

GPT-4o — $15/1M, Claude 3 Haiku $1.25

$15.00

Jan 2025

DeepSeek-V3 API — $2.19/1M, Together AI $0.80

$2.19

Jun 2025

Gemini Flash/GPT-4o-mini class — $0.60

$0.60

100× cost reduction in ~26 months. Driven by: better quantization, continuous batching, hardware (H100→B200), and competition between providers.

The Jevons paradox — falling costs don't mean falling revenue

When compute costs fall 100×, demand expands. In 1865, William Stanley Jevons showed that more efficient steam engines led to more coal consumption (not less) because cheap energy unlocked new use cases. The same applies to inference: 100× cheaper tokens means LLMs embedded in every app, agent workflow, and background process — not 100× fewer tokens consumed.

The implication for NVIDIA and TSMC: falling per-token cost → expanding token volume → more GPU demand, not less. The bull case is that inference scaling is demand-elastic in the same way cloud computing was demand-elastic to falling AWS prices. This is why NVIDIA's data center revenue has not peaked despite inference cost deflation.

5–23×

Throughput gain from continuous batching vs. static

vLLM benchmark on Llama at realistic traffic patterns

2–4×

Throughput gain from PagedAttention vs. HuggingFace

Memory waste reduction: 60% → <4%

2–3×

Latency speedup from speculative decoding

At 75%+ draft acceptance rate, ~3× tokens/sec

2×

Throughput gain: FP8 vs. BF16 on H100

INT8/FP8 Tensor Cores: 7,916 vs. 3,958 TOPS/TFLOPS

Technical quiz

1. A request completes in 5 decode steps. With static batching and a batch size of 4, the remaining 3 slots sit idle for those 5 steps while the 4th request finishes its 20-step generation. What does continuous batching do differently?

It pads short sequences to match the longest sequence length, keeping all 4 slots busy with real computation

It evicts finished sequences immediately and inserts new requests into freed slots at the next decode step, so slots are never idle between batches

It runs multiple small batches simultaneously on different GPU SMs, hiding the idle time with other compute

It predicts which requests will finish early and groups them into one batch, keeping similar-length requests together

2. PagedAttention allocates KV cache in fixed-size "blocks" mapped via a block table. What specific memory problem does this solve that pre-allocating the maximum sequence length cannot?

It allows KV cache to be stored in CPU RAM rather than GPU HBM, making it unlimited in size

It reduces the per-token KV cache size by compressing attention patterns with a learned codebook

It eliminates internal and external memory fragmentation — allocating only the blocks a sequence actually uses, so free memory stays available for new requests instead of being trapped as reserved-but-unused space

It enables the KV cache to be shared across multiple model layers, reducing total memory proportional to the number of layers

3. Speculative decoding is mathematically guaranteed to produce the same output distribution as running the target model alone. Why is this guarantee possible despite using a different (smaller, less accurate) draft model?

The draft model is a distilled version of the target model, so it always produces the same token distribution for common sequences

Rejected draft tokens are resampled directly from the target model's true probability distribution, so every accepted or resampled token comes from the target distribution — the draft model only determines how many tokens can be verified in one pass

The target model verifies draft tokens using a modified softmax that corrects for the draft model's distribution shift

The guarantee only holds when draft acceptance rate exceeds 80%; below that threshold, output quality degrades

4. FP8 quantization (W8A8) on the H100 achieves ~2× the throughput of BF16 for the same model on the same hardware. What is the primary reason for this speedup?

FP8 weights load from HBM 2× faster because each weight is half the size, doubling effective memory bandwidth for the same data

H100 Tensor Cores execute FP8 matrix multiplies at 7,916 TOPS vs. 3,958 TFLOPS for BF16 — both weights and activations participate in INT8/FP8 Tensor Core operations, doubling compute throughput while also halving memory traffic

FP8 quantization allows two model layers to be fused into one kernel, cutting the number of kernel launches in half

FP8 activations fit in the L2 cache instead of requiring HBM reads, dramatically improving cache hit rate for the attention layers

5. Inference token costs have fallen ~100× in 26 months. From an investor perspective, why does this NOT necessarily imply that NVIDIA's data center revenue will fall?

NVIDIA captures inference economics through software licensing (CUDA/TensorRT subscriptions), not hardware sales

The 100× cost reduction is offset by NVIDIA raising GPU prices proportionally to capture the cost savings

Jevons paradox: 100× cheaper inference unlocks use cases that were uneconomical at higher prices (every app, every background process), expanding total token volume faster than price falls — the same pattern as AWS cloud where falling $/compute led to more compute spend, not less

Inference cost is not the binding constraint — NVIDIA's GPU supply is the bottleneck, so lower costs don't reduce revenue because supply always sells out

Primary sources

vLLM / PagedAttention paper (Kwon et al., UC Berkeley 2023) — Section 2 (background) and Section 3 (PagedAttention) are the most important 6 pages. The motivation and the block table design are both here.
Orca: Continuous Batching (Yu et al., OSDI 2022) — The origin of continuous batching. Section 3 (the request-level scheduling insight) is the key read.
Speculative Decoding (Leviathan et al., Google 2023) — Section 2 (the sampling theorem) proves output distribution equivalence. Short paper, very readable.
vLLM GitHub — Deploy Llama 3 8B locally with one command. The fastest way to build intuition for what continuous batching feels like in practice.

What comes next

Lesson 9: Google TPU — how a systolic array works, why it differs fundamentally from a GPU, and what XLA enables that CUDA cannot easily match

Ask me anything. Good follow-ups for a tech worker: "Walk me through the block table data structure in PagedAttention and the attention kernel modification" · "How does the online softmax normalization in FlashAttention enable streaming speculative verification?" · "Implement a simple continuous batching scheduler in Python" · For an investor: "Which inference framework is most used by hyperscalers — TRT-LLM or vLLM?" · "How does disaggregated prefill-decode change the GPU mix needed for a $100M inference cluster?" · "Model the impact on SK Hynix if speculative decoding becomes universal — does HBM demand go up or down?"