Hybrid linear attention: rationing the quadratic
ai
llm
attention
inference
architecture
Subquadratic attention spent about ten years as the thing that almost worked. Every couple years someone would ship a linear-attention variant, it would lose two or three points of accuracy against a real transformer, and everyone would go back to softmax. In 2026 that story flipped, and it flipped because three labs that don't talk to each other landed on the same number.
MiniMax-01, Qwen3-Next, and Kimi Linear all ship hybrid stacks with a 3:1 ratio of linear attention blocks to full attention blocks. Three cheap layers, one expensive one, repeating. That's not a coincidence and it's not "linear attention finally got good enough." It's an admission that pure linear attention is structurally bad at one specific thing, retrieving a particular earlier token, and the fix isn't to make linear attention smarter. It's to stop asking it to do the one job it can't do, and hand that job to a small number of full attention layers instead.
You don't beat the quadratic. You ration it.
Full attention computes, for every token t, a weighted sum over every prior token:
out_t = sum over i <= t of softmax(q_t . k_i) * v_i
To serve token t you need every prior k_i, v_i sitting in memory, the KV cache, and that cache grows linearly with how far into the sequence you are. That's the O(n squared) time and the growing-memory problem everyone complains about with long context.
Linear attention starts from a simple move: drop the softmax, replace it with a kernel feature map φ applied to the keys, and the whole sum turns into something you can compute incrementally. Define a running state:
S_t = sum over i <= t of v_i * phi(k_i)^T (a d x d matrix)
out_t = S_t . phi(q_t)
S_t = S_(t-1) + v_t * phi(k_t)^T
That's it. History stops being a list of tokens you keep around and becomes a fixed-size matrix you update with an outer product. O(n) total compute, O(d squared) memory, and critically that memory is independent of how long the sequence gets. This is the whole appeal in one line: the past is compressed into a matrix instead of stored as a list.
It's also the whole problem, but we'll get there.
Plain linear attention has a design flaw: it only ever adds. It never forgets, and it never corrects a bad write. Two separate lineages fixed pieces of this, and the 2026 wave fuses them.
The DeltaNet delta rule says: don't blindly accumulate v_t * phi(k_t)^T into the state. Instead, write the error between the new value and what the state already predicts for that key. If the state already thinks key k_t maps to something, and it's wrong, correct it:
S_t = S_(t-1) + beta_t * (v_t - S_(t-1) . k_t) * k_t^T
= (I - beta_t * k_t k_t^T) . S_(t-1) + beta_t * v_t * k_t^T
That (I - beta * k k^T) term is the important part. It erases the old association at key k_t before writing the new one. It's a targeted overwrite, not a pile-up.
Mamba-2 style gating says: let a scalar or diagonal decay alpha_t between 0 and 1 shrink the retained state every step, so old memories fade at a controllable rate instead of living forever.
Fuse them and you get the recurrence that shows up, with minor notational variants, in every one of these 2026 architectures (Gated Delta Networks):
S_t = (I - alpha_t * beta_t * k_t k_t^T) . S_(t-1) + beta_t * v_t * k_t^T
alpha is how much of the old state you keep. beta is how hard you write the new one. k and v are where and what. Still a fixed-size state, still O(n) time, but now it can forget on purpose and correct itself instead of just piling associations on top of each other. That's the difference between "linear attention" as a 2021 curiosity and Gated DeltaNet as the thing actually load-bearing in frontier models right now.
Here's the step as code, small enough to run in your head:
import numpy as np
def gated_delta_step(S, k, v, alpha, beta):
kk = np.outer(k, k)
forget_and_correct = np.eye(S.shape[0]) - alpha * beta * kk
write = beta * np.outer(v, k)
return forget_and_correct @ S + writeWrite key A, write key B, then write A again with a different value, and you'll see the A slot get cleanly overwritten while B is untouched. That's the delta rule doing its job: a targeted edit, not a blind add.
Per layer, at sequence length n, full attention's KV cache costs:
M_cache(n) = 2 * n * d_kv * bytes_per_value
Linear (or Gated DeltaNet) memory costs:
M_state = d^2 * bytes_per_value
One of these is a function of n. One isn't. The cache line rises forever; the state line is flat. Set them equal and solve for the crossover:
n_cross = d^2 / (2 * d_kv)
Past a fairly modest context length, the KV cache dwarfs the constant state, and the gap keeps widening without bound as n grows. That's not a marginal win. That's the difference between "long context is expensive" and "long context is free, memory-wise."
def kv_cache_bytes(n, d_kv, bpv=2):
return 2 * n * d_kv * bpv
def state_bytes(d, bpv=2):
return d * d * bpv
d, d_kv = 4096, 128
crossover_n = (d ** 2) / (2 * d_kv)Run that with realistic d and d_kv and the crossover lands well inside typical chat-context lengths. Past it, the flat line is why long-context serving stops being a memory problem, at least for the layers that are linear.
Here's the catch, and it's the same property that made the memory win possible.
The state S is a d x d matrix holding a superposition of every key-to-value association written into it so far. To read back a key k, you compute S . k. That works fine when you have a handful of distinct keys. It stops working once you've written more associations than the matrix has rank to hold cleanly, roughly d of them. Past that, new writes start interfering with old ones. The delta rule helps, because it overwrites the same key cleanly instead of piling on top, but it doesn't change the fact that the matrix has finite rank. Write a needle fact early in a long context, keep writing other stuff for thousands of tokens, and eventually that early write gets crushed by everything that came after.
You can see this with a toy that has nothing to do with a real transformer, and it's honest to say so:
import numpy as np
def write_and_read(d, m, needle_index=0):
S = np.zeros((d, d))
keys = [np.random.randn(d) for _ in range(m)]
values = [np.random.randn(d) for _ in range(m)]
for k, v in zip(keys, values):
S += np.outer(v, k) / np.linalg.norm(k)
read = S @ keys[needle_index]
target = values[needle_index]
cos_sim = read @ target / (np.linalg.norm(read) * np.linalg.norm(target))
return cos_sim
for m in [4, 32, 128, 512, 4096]:
print(m, write_and_read(d=64, m=m))Sweep m up past d and recall of the first-written pair collapses. Do the same "read the first thing back" test with real softmax attention over the same m pairs and it stays near perfect, because softmax attention doesn't compress anything. It re-reads the actual stored k_i, v_i pairs, content-addressable, lossless within its window. O(n squared), but exact.
So the two failure modes are opposite and neither is negotiable:
This is the actual insight behind the hybrid, and it's simpler than the math makes it look: softmax attention is content-addressable and the linear layers structurally can't be. A linear layer compresses everything into one matrix and hopes the important stuff survives the compression. A full attention layer doesn't compress at all, it just looks things up.
So you stop trying to make the linear layers remember better, and you give the stack a small number of layers whose entire job is exact lookup. The three linear layers carry the bulk of the sequence processing cheaply. The one full attention layer per group is the associative memory the other three structurally cannot be.
Total cost across a stack with a fraction p of full attention layers is roughly:
cost ~= (1 - p) * O(n) + p * O(n^2)
The quadratic term still dominates asymptotically, but with a small constant in front of it if p is small. Push p down and you get cheaper, but the retrieval that only full attention layers can do degrades, because there's less of that lookup capacity distributed through the stack. Push p up and retrieval is fine but you've given back the savings that made this worth doing in the first place.
3:1, one full attention layer per three linear ones, is where MiniMax-01, Qwen3-Next, and Kimi Linear independently landed, and it's worth being precise about what that number is and isn't. It's the empirically converged-upon sweet spot (Hybrid Linear Attention Done Right), not a theorem with a clean derivation behind the specific 3. Different teams tune it slightly and interleave the layers differently. Treat "3:1" as a budget the field settled on, not a constant of nature.
Inference cost and long-context economics are not abstract for me. The KV cache is the concrete thing that makes long-context chat expensive to serve, and "constant-memory state vs a cache that grows linearly with every token you keep in context" is the entire ballgame for what it costs to run something at scale. A flat memory line means long-context stops being a capacity-planning problem for whichever fraction of a stack is linear. That's not a research curiosity, that's a line item.
The recurrence above has minor notational variants across papers depending on where alpha and beta attach. The form here matches the Gated Delta Networks paper; treat it as a canonical form, not the universal one.
3:1 is empirical, not derived from first principles. Some architectures interleave differently or tune the ratio per layer group. "Converged-upon sweet spot" is the honest description, "optimal ratio" is not.
The "capacity is roughly d" intuition for the state is a rank argument, not a tight theorem. Gating and the delta rule push effective capacity further than naive accumulation would suggest, they don't remove the bound.
Full attention is lossless within its context window, not globally. Give it more tokens than fit in the window and it forgets too. The hybrid's advantage is about behavior inside a fixed context length, not about escaping context limits altogether.
A fixed-size state is the whole win and the whole catch, and you can't get one without the other. It's what makes memory flat as context grows, and it's exactly why exact recall is structurally impossible from a linear layer alone. So you keep one full attention layer per three to be the associative memory the linear layers can't be, and you get most of the cost savings anyway because that one layer is a minority of the stack.
If you build the needle-in-a-haystack toy yourself and watch a pure-linear stack lose the needle while a 3:1 hybrid catches it every time, you will never again read "linear attention" as a synonym for "free attention." It isn't free, it's a trade, and the whole architecture story of 2026 is three labs figuring out independently how to make that trade on purpose instead of by accident.
Sources: