L22 said decode re-reads every weight and the KV cache grows with context. Two architectural moves attack exactly those two costs — MoE breaks "read every weight," SSMs break "KV grows forever." Each rewrites the compute–memory ratio, and therefore which hardware (and which thesis name) wins inference.
Builds on: L7 (transformer), L21–L22 (inference cost)Skill: predict how architecture shifts hardware demand
You now understand inference at the metal: decode is memory-bound because each token re-streams the whole model, and the KV cache is what fills your HBM (L22). Architects know this too — and the two hottest model families of the last two years are direct attacks on those two costs. Mixture-of-Experts (MoE) attacks "re-read every weight." State Space Models (SSMs / Mamba) attack "KV grows with context." Understanding them lets you do the thing the mission is about: read an architecture announcement and predict which hardware bottleneck — and which company — it favors.
Warm-up — Retrieve L22 First
retrievalL22
Before reading on, answer from memory: in the decode phase, why is the GPU memory-bandwidth-bound rather than compute-bound — and what data structure grows with context length and caps how many users fit on a GPU?
Answer
Decode generates one token at a time, and each token requires re-reading all model weights from HBM while doing very little math → starved for bandwidth, not FLOPS. The KV cache grows linearly with context (and batch), dwarfs the weights at long context, and caps concurrent users. MoE attacks the first; SSMs attack the second.
Core thesis:MoE decouples a model's compute from its size: total parameters explode (more knowledge) while only a small subset activate per token (cheap compute). But all experts must still live in HBM, and routing adds all-to-all networking — so MoE shifts the bottleneck from compute toward memory-capacity and interconnect. SSMs replace attention's growing KV cache with a fixed-size recurrent state, making long context O(n) and cheap — which, if adopted, relieves the very HBM scarcity that underpins the SK Hynix bull case. Architecture choices are hardware-demand bets.
01 — The Dense Baseline
Why "Every Token Uses Every Weight" Is the Problem
A classic ("dense") transformer runs every parameter for every token. That's the L22 decode tax: a 400B dense model streams 400B weights from HBM per token. Worse, making a dense model smarter means making it bigger, which raises both the compute and the memory-read per token in lockstep. The cost of knowledge is welded to the cost of inference. MoE breaks that weld.
02 — Mixture of Experts
Many Experts, Few Awake
MoE replaces the dense feed-forward block with N expert sub-networks plus a small router that, for each token, picks the top-k experts to run. The rest stay asleep [Wolfe]:
Dense FFN vs. MoE layer — the router wakes only top-k experts
Mixtral activates 2 of 8 experts; DeepSeek-V3 activates 8 of 256 routed experts (plus shared always-on experts). The router is trained to send tokens where they're handled best. [DeepSeek MoE]
compute / token ∝ ACTIVE params | HBM capacity ∝ TOTAL params
The whole point: these two numbers separate. DeepSeek-V3 = 671B total but only ~37B active (~5.5%); Mixtral-8x7B = 46.7B total, ~13B active.
Play with the separation — set total and active params and watch what each axis costs:
MoE: Compute vs. Memory Separation
A simplified model: compute/token tracks active params; HBM capacity tracks total params. (Defaults ≈ DeepSeek-V3.)
Compute / token
like 37B
dense-equivalent FLOPs
HBM capacity
like 671B
must hold all experts
Compute saving
18×
vs. dense of same size
03 — MoE's Hardware Shadow
The Bottleneck Moves to Memory and the Network
MoE is why frontier-quality inference got dramatically cheaper (DeepSeek served a near-frontier model at a fraction of dense-model cost) [MoE economics]. But "cheaper compute per token" does not mean "cheaper to run." It relocates the cost:
HBM capacity goes up, not down. You must hold all experts in memory even though most sleep each token. A 671B MoE has the memory footprint of a 671B dense model — the L22 KV/weight memory wall, intensified.
Networking becomes load-bearing. In distributed serving, experts are spread across GPUs (expert parallelism), so every token triggers all-to-all communication to reach its chosen experts and return — exactly the scale-out interconnect from L13. [MoE on GPU cloud]
Compute utilization gets harder. Routing is dynamic and uneven (load-balancing problem), so keeping all GPUs busy is a serving-software challenge.
Read-through: MoE shifts the binding constraint from raw FLOPS toward memory capacity + interconnect bandwidth. That is bullish for HBM capacity (SK Hynix, AMD's capacity wedge) and for scale-up/scale-out networking (NVLink, Broadcom Ethernet/switching) — and it's why hyperscalers love MoE: it converts the expensive, scarce resource (compute) into a cheaper-per-token design while leaning on memory and networking they're already buying.
04 — State Space Models (Mamba)
Killing the Growing KV Cache
The other great cost in L22 was the KV cache: attention must look back at every previous token, so memory grows linearly with context and compute grows ~quadratically. SSMs attack this at the root. Instead of attending to all past tokens, a Mamba layer keeps a fixed-size recurrent state that summarizes the past, updated token by token — like a control-theory system [Mamba primer]:
Property
Attention (Transformer)
SSM (Mamba)
Compute vs. length
~O(n²)
O(n) — linear
Memory for context
KV cache grows with n
Fixed-size state (no growth)
Long-context cost
Brutal (L22 calculator)
Cheap, ~constant memory
Throughput
Baseline
Up to ~5× at long sequences
Weakness
—
Weaker exact recall / in-context lookup
The catch: a fixed-size state must forget, so pure SSMs are worse at precise recall (e.g. "what was the 3rd item in that list 200k tokens ago?"). The winning answer so far is hybrids: mostly Mamba layers for cheap sequence mixing, with a few attention layers for exact lookup. Jamba (Transformer-Mamba-MoE, ~1:7 attention:Mamba) runs 256K-token context with a ~9 GB KV cache — an order of magnitude smaller than a pure transformer. [hybrid LLMs]
05 — Investment Synthesis
Architecture as a Hardware-Demand Bet
MoE → bullish HBM capacity + networking
Total params (HBM footprint) keep rising even as compute/token falls, and expert parallelism makes all-to-all interconnect load-bearing. Net: SK Hynix (capacity), AMD (capacity wedge), Broadcom (Ethernet/switching) and NVDA (NVLink) all benefit. MoE is a tailwind for the memory + network layers, not just compute.
MoE → the deflation/demand engine
By cutting compute/token ~10–20×, MoE slashes serving cost (L21) → cheaper tokens → far more tokens (Jevons). This is a structural answer to the "AI capex bubble" bear: efficiency gains expand usage, sustaining hardware demand rather than shrinking it.
SSMs → a genuine SK Hynix falsifier
If SSM/hybrid models make long context cheap with a tiny, fixed KV cache, they directly attack the "HBM is scarce → pricing power" thesis (L12). This is the concrete mechanism behind SK Hynix's "a memory-architecture shift reduces HBM content per accelerator" falsifier. Watch hybrid adoption at the frontier.
The net is nuanced, not one-directional
MoE pushes toward more memory/networking; SSMs push away from KV/memory. The frontier (Jamba) combines both. Don't extrapolate a single architecture into a clean thesis — track the mix, because the compute:memory:network ratio it implies is what actually sets demand across NVDA / AMD / SK Hynix / Broadcom.
1. In a Mixture-of-Experts model, compute per token scales with the:
Total parameter count, since all experts must be loaded
Active (top-k) parameters, since only chosen experts run
Number of users currently connected to the serving cluster
Length of the output, regardless of which experts are picked
2. Despite activating few experts per token, an MoE model's HBM footprint is set by:
Only the active experts, so memory needs drop sharply too
The router alone, which is tiny relative to the experts
Total parameters, since all experts must sit in memory
The KV cache only, as weights are streamed from disk
3. Distributed MoE serving makes networking load-bearing because:
Experts spread across GPUs need all-to-all token routing
Each user must connect directly to a dedicated expert GPU
The router runs on a remote server far from the experts
MoE models cannot use HBM and must fetch over the network
4. State space models (Mamba) reduce long-context cost mainly by:
Keeping a fixed-size state instead of a growing KV cache
Running every parameter on every token like a dense model
Storing the entire context permanently on the host CPU
Increasing attention to quadratic cost for better recall
5. Pure SSMs are usually paired with a few attention layers (hybrids) because:
A fixed-size state forgets, weakening exact long-range recall
Attention layers are required to load the model into memory
SSMs cannot run on GPUs without an attention co-processor
Hybrids are needed only to make the model train faster overall
6. For the SK Hynix thesis, broad SSM/hybrid adoption would be a:
Bull confirmation, since SSMs need far more HBM per chip
Falsifier, since cheap fixed KV reduces HBM content per accelerator
Neutral event, because architecture never affects memory demand
Bull confirmation, since SSMs make every model strictly larger
From your instructor: The frame to keep — MoE separates compute (active params) from memory (total params), shifting the bottleneck toward HBM capacity + networking; SSMs replace the growing KV cache with a fixed state, relieving the long-context memory wall. Both are architectural moves aimed squarely at the L22 cost structure, and each redirects hardware demand. Ask me anything: how the router is trained, why all-to-all is the MoE bottleneck, how a selective-scan SSM actually updates its state, or why hybrids beat pure Mamba.