Speculative decoding and attention drift
ai
llm
inference
speculative-decoding
transformers
Speculative decoding sounds like a scam. A small cheap model guesses the next several tokens, a big expensive model checks them, and you get 2-6x the speed for free, with output that is bit-identical to what the slow model would have produced on its own. That claim is true, and it is provable in about ten lines of algebra. The part people get hand-wavy about isn't the proof, it's why pushing the trick deeper eventually breaks it, and the answer isn't "the small model got dumber." It's that the small model stopped paying attention to the prompt.
I run b3iq at B3, where draft and target pairing across an open-weight fleet is a real latency lever, not a theory exercise. "How deep can I let the drafter run before acceptance falls off a cliff" is a tuning knob with money attached. So this is the post where I derive why the free lunch is actually free, and then explain the mechanism nobody animates: attention drift, named by EAGLE-3.1 in May 2026.
The protocol is three lines, and the trick is in the second one:
k tokens, one at a time, autoregressively. Cheap.k drafted tokens. Expensive, but you only pay for it once.k drafted tokens and accept or reject each one against what the target would have said.Step 2 is the part that looks like magic until you remember the target model is causal. Feed it [prompt, d1, d2, ..., dk] and the causal mask means every prefix gets its own set of next-token logits in the same pass: p_t(. | prompt), p_t(. | prompt, d1), all the way through p_t(. | prompt, d1...dk). One pass, k+1 distributions. That's the entire source of the speedup: you run the big model once and read off k+1 answers it was computing anyway, because a transformer forward pass computes logits for every position regardless of whether you asked for them.
Here's the part everyone believes without checking. For a drafted token x with draft probability p_d(x) and target probability p_t(x):
accept x with probability min(1, p_t(x) / p_d(x))
if rejected, resample x' from p_resid, where
p_resid(z) = (p_t(z) - p_d(z))_+ / sum_z (p_t(z) - p_d(z))_+
(.)_+ just means "clip at zero", since p_t - p_d can be negative.
Claim: the token you end up emitting, after accept/reject, is drawn exactly from p_t. Not approximately. Exactly.
Proof. The probability of finally emitting some token z has two ways to happen: you drafted z and it got accepted, or you drafted something else, it got rejected, and the resample landed on z.
P(emit z) = p_d(z) * min(1, p_t(z)/p_d(z)) # drafted z, accepted
+ P(reject) * p_resid(z) # drafted something else, resampled z
The first term simplifies nicely: p_d(z) * min(1, p_t(z)/p_d(z)) is just min(p_d(z), p_t(z)). The overall rejection probability is 1 - sum_x min(p_d(x), p_t(x)), which, since probabilities sum to 1, works out to sum_x (p_t(x) - p_d(x))_+. Multiply that by p_resid(z) and the normalizing denominator cancels, leaving (p_t(z) - p_d(z))_+.
Add the two terms:
P(emit z) = min(p_d(z), p_t(z)) + (p_t(z) - p_d(z))_+ = p_t(z)
That last step is just casework: if p_t(z) >= p_d(z), the min is p_d(z) and the clipped term is p_t(z) - p_d(z), which sum to p_t(z). If p_t(z) < p_d(z), the min is p_t(z) and the clipped term is zero. Either way it lands exactly on p_t(z).
This is the Leviathan et al. and Chen et al. result from the original speculative decoding papers, and every EAGLE variant inherits it unchanged. The target's sampling distribution is preserved token for token. "Lossless" here means identical in distribution, same sampling law, same RNG, same temperature, not that two runs on different hardware will produce the same bits. That's a separate and much messier claim about floating point nondeterminism, and it's not what's being claimed here.
Acceptance rate has a clean closed form. The probability a single drafted token survives, averaged over the drafter's own distribution, is:
alpha = E_x~p_d[min(1, p_t(x)/p_d(x))] = sum_x min(p_d(x), p_t(x)) = 1 - TV(p_d, p_t)
TV is total variation distance between the draft and target distributions. Acceptance is just one minus how different the two models are at that position. A drafter that mimics the target well gets accepted, a drafter that drifts away from it doesn't.
With acceptance alpha held constant across a chain of k drafted tokens, the expected number of accepted tokens per target pass is a geometric sum:
E[accepted] = (1 - alpha^(k+1)) / (1 - alpha)
Let c be the cost of one draft step relative to one target step (small, since the drafter is small). Expected speedup per target-model-equivalent of work is roughly:
E[speedup] ≈ (1 + sum of acceptance terms) / cost_ratio
The thing worth sitting with: the lever is alpha, not k. Cranking k up only helps as long as alpha^k hasn't decayed to nothing. Past that point you're paying for drafted tokens that get rejected anyway, and if alpha actually falls as k grows, which is exactly what happens with drift, deeper drafting makes things worse, not better.
A single drafted chain bets everything on one sequence of guesses. EAGLE-2 and EAGLE-3 instead propose a tree of candidates at each step, branching on the drafter's own confidence, and the target verifies the whole tree in one pass by extending the causal mask to cover every branch. If the top guess is wrong but the second-ranked branch happens to match the target, you still bank an accepted token instead of discarding the whole step. More candidates per position raises the effective acceptance rate without needing a better drafter, which is why trees are now the default over chains in production speculative decoding stacks (NVIDIA's overview covers this well).
So if trees and acceptance are the whole story, why doesn't everyone just crank depth and branching forever? Because the drafter is a small transformer with its own attention pattern, and transformers lean on something called an attention sink, usually the first or BOS token, as a kind of no-op reservoir for attention mass that doesn't want to go anywhere specific.
At shallow draft depth the drafter is mostly attending to the prompt, including that sink token, and its output distribution tracks the target closely. As draft depth j grows, the drafter is increasingly conditioning on tokens it itself emitted a moment ago, d1 through d(j-1). Attention mass migrates off the sink and onto the drafter's own recent output. The drafter starts modeling its own drift instead of the target's actual distribution.
In the terms from above: TV(p_d^(j), p_t^(j)) grows with j, so alpha_j = 1 - TV falls with j. The constant-alpha assumption behind the speedup math quietly stops being true, and acceptance collapses right where you'd want depth to be paying off most. EAGLE-3.1 is the paper that put a name on this in May 2026, calling it attention drift (MarkTechPost's writeup), and framing it as a structural problem with where attention goes, not a capacity problem with the drafter's weights.
Here's a small simulation of the shape, decaying alpha against a flat baseline, to make the collapse concrete:
def accepted_tokens(alphas):
"""alphas: list of per-depth acceptance rates (alpha_1, alpha_2, ...)."""
prod = 1.0
total = 0.0
for a in alphas:
prod *= a
total += prod
return total
depths = range(1, 13)
# flat drafter: alpha stays near 0.85 regardless of depth
flat = accepted_tokens([0.85] * 12)
# drifting drafter: alpha decays as depth grows
drift_alphas = [0.85 * (0.92 ** j) for j in range(12)]
drift = accepted_tokens(drift_alphas)
print(flat, drift)Run that out to depth 12 and the flat-alpha drafter's expected accepted tokens keeps climbing toward its geometric ceiling, while the drifting drafter's curve peaks somewhere in the middle depths and then bends back down, because each additional drafted token is less likely to survive than the one before it. That peak-then-fall shape is the whole story in one plot. It's a simulation tuned to reproduce the reported collapse, not a measured trace pulled from the paper, worth saying plainly.
EAGLE-3.1's fix is structural rather than a bigger drafter. Instead of letting the drafter condition only on its own most recent hidden state, which is where the self-referential drift creeps in, it fuses features from three depths of the target model: early layers (roughly syntactic structure), mid layers (semantic content), and late layers (close to the output distribution). Feeding the drafter this blended signal keeps it anchored to what the prompt looks like at multiple levels of abstraction, instead of just extrapolating from its own last few guesses.
The effect on the attention pattern is what you'd hope: with fusion on, attention mass at depth stays closer to the sink instead of sliding onto the drafter's own tail. Acceptance holds up at depths where the unfused drafter was already collapsing. That's the mechanism: a change to what the drafter is allowed to look at.
Running a fleet of open-weight models with different draft/target pairings, the acceptance-versus-depth curve is the actual thing you're optimizing, not draft model size or target model size in isolation. A drafter that's 10x cheaper but drifts hard by depth 6 can lose to a slightly more expensive drafter that holds acceptance to depth 10. The naive instinct is "make the draft model smaller and faster." The better instinct, per EAGLE-3.1's framing, is "make the draft model harder to distract from the prompt."
"Lossless" and "bit-identical" throughout this post mean identical in distribution, the same sampling law given the same random seed and temperature as the target alone, which is the standard and correct reading of the Leviathan/Chen result. It is not a claim that two hardware configurations will produce identical floating point outputs; that's a different and separate nondeterminism problem.
The 2-6x speedup range is workload and pairing dependent. It moves with draft quality, the cost ratio between draft and target, and how predictable the domain is (code completion accepts deeper than open-ended creative text, for instance). Treat it as a typical range from the literature, not a guarantee for any specific pair you run.
"Attention drift" and "tri-layer feature fusion" are EAGLE-3.1's 2026 framing and terminology, attributed to that paper and the MarkTechPost coverage of it. The acceptance-vs-depth simulation above is illustrative, tuned by hand to reproduce the shape of the reported collapse, not a measured trace extracted from the paper itself.
The losslessness is settled math: min(1, p_t/p_d) plus a residual resample, and you draw exactly from the target's distribution, no asterisk. The open question for anyone actually running inference at scale is acceptance at depth, and EAGLE-3.1's answer is that drift is an attention problem, not a capacity problem. Re-anchor the drafter on syntax, semantics, and output-distribution features together, and it stays accepted deeper than a drafter left to drift onto its own tail.
If you measure your own acceptance-versus-depth curve on whatever draft/target pair you're running and find the point where it bends, I'd like to see the plot.
Sources: