The KV-cache architecture wars
ai
llm
architecture
inference
kv-cache
Look at a 2026 open-weights spec sheet and the headline number isn't params anymore. It's how much context the model serves and how much memory that costs you to serve it. Nobody's flexing 8B vs 70B this year. They're flexing "128K context, batch 8, fits on one card." That's the whole fight now, and it's a fight over one formula.
I run b3iq at B3, where we route requests across an open-weight fleet, and "can this node hold 128K context at batch 8" is a scheduling constraint I hit every week, not a footnote in a paper. So when I read the 2026 architecture surveys and everyone's got a different clever name for their attention variant, I stopped trying to memorize the taxonomy and just wrote down the formula everyone's actually optimizing. It's short:
cache = 2 · L · n · d_kv · bytes
Every architecture you've heard of in the last two years, GQA, MLA, Gemma's cross-layer sharing, DeepSeek's compressed attention, is an attack on exactly one letter in that equation. Learn the formula and the marketing collapses into four moves.
Start from what attention actually has to remember. For every token, at every layer, the model computes a Key vector and a Value vector and it has to keep both around for every future token that attends back to this one. That's the whole cache, there's nothing more mystical in it than "store two vectors per token per layer."
So the byte cost is:
KV_bytes = 2 · L · n · d_kv · bytes · B
▲ ▲ ▲ ▲ ▲ ▲
K&V layers seq head-dim dtype batch
Walking it:
2 is because you're storing both K and V, alwaysL is the number of transformer layers, each with its own cachen is sequence length, every token you've processed so fard_kv is the total K/V dimension stored per token per layer. For plain multi-head attention this is n_heads · d_head, which equals d_modelbytes is your dtype, 2 for fp16/bf16, 1 for fp8B is batch size, because every concurrent request gets its own cachePlug in Llama-3.1-8B with full multi-head counting: L=32, d_model=4096 so d_kv=4096, n=131072 (128K), bytes=2, B=8.
KV ≈ 2 · 32 · 131072 · 4096 · 2 · 8 bytes ≈ 5.5e11 bytes ≈ 550 GB
That's not the real number, and I want to be honest about why I'm showing it: it's the teaching number. Llama-3.1-8B doesn't actually store full multi-head K/V, it already ships with grouped-query attention, 8 KV heads instead of 32. Once you apply that (more on GQA in a second), the real cache at 128K context and batch 8 lands around 130-140 GB. That's the number people actually quote, and it's still more memory than the model's weights. An 8B model in fp16 is roughly 16 GB. Its cache at long context and modest batch is 8-9x the size of the model itself. That's the whole crisis in one sentence: at long context, the cache eats the GPU, not the weights.
Every technique below is somebody looking at that 130 GB and deciding which letter to attack.
d_kv by grouping headsMulti-head attention gives every query head its own dedicated KV head. d_kv = n_heads · d_head, full stop, no sharing. It's expensive because you're storing a distinct K and V subspace for every one of your, say, 32 attention heads, even though a lot of those subspaces end up doing similar work.
Grouped-query attention's move: keep all your query heads, but have groups of them share a single KV head. Define group size g = H_q / H_kv. MHA is the special case g=1. GQA sets g > 1, so:
d_kv_gqa = H_kv · d_head = d_kv_mha / g
Llama-3.1 uses g = 32/8 = 4, so its cache is a quarter of the full-MHA number. That's the drop from 550 GB down to ~135 GB in the example above. The cache shrinks by exactly the group factor, linearly, because you've made d_kv smaller.
The cost: fewer distinct KV subspaces means less head diversity. Four query heads are now reading off the same K/V, so whatever specialized attention pattern one of those heads wanted, it's sharing a lookup table with three others. Models trained with GQA from scratch mostly recover the quality, models retrofitted with GQA after MHA pretraining pay more of a tax. Either way, GQA is buying cache by literally deleting distinct heads. That's the ceiling.
d_kv differently, without deleting headsDeepSeek's move, multi-head latent attention, looks at the same problem and refuses the tradeoff. Instead of throwing away distinct heads (GQA's move), it compresses the hidden state into a shared low-rank latent vector c of dimension d_c, where d_c is much smaller than d_kv. You cache the latent, not the per-head K/V. When it's time to attend, you reconstruct each head's actual K and V from the latent via a per-head up-projection, folded straight into the attention matmul.
cache_mla ∝ d_c (cache the latent, not the heads)
compression ratio ≈ d_kv / d_c
This is the part people gloss over and it's the actual insight: MLA keeps head diversity because the reconstruction step still produces genuinely distinct K/V per head. It just refuses to store them distinctly. GQA throws away information by sharing a small number of literal KV heads across many query heads. MLA keeps the full head diversity mathematically, and only compresses the storage, betting that the per-head K/V lives on a shared low-rank manifold most of the time (it mostly does, that's why it works).
This is why MLA sits above GQA on the quality-per-byte frontier in basically every writeup I've seen, including Raschka's 2026 architecture survey. It's not a free lunch though: you're paying compute to reconstruct heads at inference time that GQA never bothered computing in the first place. Memory-for-compute is the actual trade, and it's a good one right now because memory bandwidth, not FLOPs, is usually the bottleneck at long context.
L, not d_kvEverything above lives inside one layer's cache. Gemma's move is orthogonal: attack the layer count itself. If later layers just reuse an earlier layer's KV instead of computing and storing their own, your effective layer count for caching purposes drops:
L_eff = ρ · L + (1 - ρ) · L_shared < L
where ρ is the fraction of layers still computing their own KV and the rest point at a shared cache from a layer group. Roughly halving that (share every other layer, or share within blocks) gets you close to half the cache with a fraction of the quality cost of halving heads. For a small E2B-class model this is on the order of ~2.7 GB saved at 128K context, that specific figure is model-specific and won't transfer, but the mechanism, "cache is a function of how many independent layers actually need their own memory," generalizes to anything.
This is the axis people miss because GQA and MLA get all the press. L and d_kv are independent terms in the formula. You can do GQA and cross-layer sharing on the same model, they don't compete for the same bytes.
n itselfThe newest move, and the most aggressive: what if the sequence length term is compressible too? DeepSeek's reported approach for a V4-Pro-class model with heavily-compressed attention takes windows of tokens, learned-pools or latent-compresses them, and caches one KV entry per window instead of per token:
n_eff = n / w (w ≈ 128 in the heavily-compressed regime)
Reported figures put this around 128 tokens compressed into 1 KV entry, yielding roughly 27% of single-token attention FLOPs and ~10% of the KV cache versus their previous generation, measured at a 1M-token context. I want to flag clearly: those numbers are reported and partly reverse-engineered from 2026 disclosures, not a published spec I've verified myself. Credible, not gospel. Cite it accordingly and don't quote it like a datasheet.
The cost here is the real one: this is lossy. Fine-grained, token-level recall inside a compressed window degrades, because 128 tokens are now sharing one memory slot instead of 128. The mitigation everyone converges on is hybrid: keep some layers or some recent window at full resolution, compress only the deep history. You're trading precision on old tokens for the ability to hold a million of them at all.
The reason this whole "war" is actually a design space and not a ladder is that the four moves hit independent terms:
cache = 2 · L_eff · n_eff · d_kv_reduced · bytes · B
└cross-layer┘ └compress┘ └GQA/MLA┘
Start at the ~130 GB Llama-3.1-8B baseline (GQA already applied). Add MLA-style compression on top of the head dimension, that's another few-x on d_kv. Add cross-layer sharing, roughly halve L_eff. Add windowed compression on the deep-history layers, divide n_eff by your window size on those layers. Multiply the factors and 130 GB is a low single-digit number of GB. That's not hand-waving, it's just multiplying independent reductions, and it's exactly the kind of thing worth writing as a real function instead of eyeballing:
def kv_bytes(L, n, d_kv, bytes_per_elem=2, B=1):
return 2 * L * n * d_kv * bytes_per_elem * B
def gqa(d_kv_mha, g):
return d_kv_mha / g
def mla(d_kv, d_c):
# cache scales with the latent, not d_kv
return d_c
def cross_layer(L, rho):
# rho = fraction of layers keeping their own KV
return rho * L + (1 - rho) * 1 # shared layers cost ~1 layer's worth
def compress(n, w):
return n / w
baseline = kv_bytes(L=32, n=131072, d_kv=gqa(4096, g=4), B=8)
print(f"{baseline / 1e9:.1f} GB") # ≈ 137.4 GB, the real Llama-3.1-8B number
stacked = kv_bytes(
L=cross_layer(32, rho=0.5),
n=compress(131072, w=128),
d_kv=mla(gqa(4096, g=4), d_c=256),
B=8,
)
print(f"{stacked / 1e9:.2f} GB") # low single digitsThat's the entire post as code. Four functions, one baseline, and a number that falls off a cliff when you compose them, because the terms don't fight each other for the same bytes.
None of this is free and I don't want the calculator to imply it is. Rough, benchmark-dependent, illustrative-not-exact:
lever cache reduction quality debit (rough)
GQA (g=4) 4x small, mitigated by from-scratch training
MLA several x on d_kv smallest of the group, costs compute not quality
cross-layer (ρ=.5) ~2x small, layer-group dependent
compression (w=128) ~w on those layers moderate to large, fine-grained recall degrades
Stack all four and you're buying a 30x-ish smaller cache at a real but bounded quality cost, and the cost isn't uniform across the levers. MLA is closest to free (it costs FLOPs, not accuracy). Aggressive windowed compression is where you actually feel it. The frontier isn't flat, some moves dominate others, and knowing which is which is the actual skill here, not memorizing acronyms.
The 550 GB figure above is a teaching number, full multi-head counting on a model that doesn't actually run that way. The real, GQA-adjusted Llama-3.1-8B cache at 128K and batch 8 is the ~130-140 GB number, and that's the one worth quoting. Don't let the 550 escape into a tweet as if it's a real deployment cost.
The DeepSeek-V4-Pro compression numbers (128 to 1, ~27% FLOPs, ~10% cache at 1M context) are reported figures from 2026 disclosures, credible but not independently verified specs. Treat them as directionally right, not as ground truth to cite in a paper.
The "~2.7 GB saved at 128K" cross-layer figure is specific to one small E2B-class model. The mechanism, effective layer count dropping when layers share KV, generalizes fine. The number does not.
MLA's cache win costs compute at inference, the per-head reconstruction is extra FLOPs folded into the attention matmul. It's a genuine memory-for-compute trade, not a free win, and whether it's worth it depends on whether you're bandwidth-bound or compute-bound on your hardware.
GQA grouped the heads. MLA compressed them without deleting them. Gemma shared them across layers. DeepSeek compressed the sequence itself. Four moves, four orthogonal terms in one formula, and they stack. The "architecture war" people write breathless posts about is really just a Pareto front, and the winners aren't the loudest names, they're the levers that buy cache without buying quality loss.
I schedule against exactly this tradeoff on b3iq every day: given a node's memory, what's the largest context and batch I can fit, and which lever do I need to pull to fit one more request. Write the four functions above, plug in your own model's L, d_kv, and target context, and find the config that fits your context on the GPU you actually have. That's the whole exercise. The formula was never the hard part, deciding which letter to attack is.
Sources: