Speculative Decoding and the Art of Fast LLM Inference

Generating one token at a time wastes most of your GPU. Speculative decoding lets a model verify several guessed tokens in a single forward pass without changing what it outputs. This is the idea, the math that proves it is exact, and the rest of the inference toolkit it sits inside.

Why Decoding Is Slow

A transformer generates text one token at a time. To produce token $t$, it needs token $t-1$ as input. That dependency is the whole problem: you cannot compute token 50 until you have committed to token 49, so generation is an inherently serial loop of forward passes.

Each forward pass in that loop is brutally inefficient. With a KV cache, generating one new token means multiplying a single token's worth of activations against the full weight matrices of the model. You load tens of gigabytes of weights from GPU memory, use them for one token, and throw them away. The arithmetic is trivial; the memory traffic is enormous.

Intuition: Decoding is memory-bound, not compute-bound. The GPU spends almost all its time waiting for weights to arrive from HBM, and almost none doing math. A modern GPU can do hundreds of floating-point operations in the time it takes to fetch one number from memory. During decode, most of that compute sits idle.

This is the key asymmetry that speculative decoding exploits. The cost of a forward pass is dominated by reading the weights, which you pay whether you push one token through or fifty. Pushing 50 tokens through in one pass costs almost the same wall-clock time as pushing one, because the bottleneck is memory bandwidth, not the matrix multiplies.

Why one token wastes the GPU $$\text{Arithmetic intensity} = \frac{\text{FLOPs}}{\text{bytes moved}} \approx \frac{2 \cdot d}{2 \cdot d} = O(1) \;\text{per weight, per token}$$

For a single token, each weight is loaded once and used in one multiply-add: arithmetic intensity near 1. GPUs need intensity in the hundreds to saturate their compute units. If you could process $k$ tokens per weight load, you would multiply your useful work by $k$ for nearly free. Batching does this across different requests. Speculative decoding does it across future tokens of the same request.

The Core Idea: Guess and Verify

The expensive model, the one you actually want to sample from, is called the target model. The trick of speculative decoding is to bring in a cheap draft model that guesses the next several tokens quickly, then have the target model check all those guesses in a single forward pass.

The draft model is small, so running it 5 times is cheap. The target model runs once over all 5 guesses at once, scoring them in parallel. If the guesses are good, you accept several tokens from one expensive forward pass instead of one. If they are bad, you fall back gracefully and lose almost nothing.

Intuition: Reading is faster than writing. It is hard to write the next sentence of a document one word at a time, but easy to read a proposed sentence and say "yes, that is what I would have written, up until this word." The target model becomes a fast proofreader of the draft model's quick guesses.

The non-obvious part, the result that made this technique matter, is that you can do this without changing the output distribution at all. The tokens you emit are sampled from exactly the same distribution as if the target model had generated them alone, one at a time. Speculative decoding is not an approximation. It is a pure speedup.

The Algorithm, Step by Step

Let the target model define the true distribution $p(x)$ over the next token, and the draft model define $q(x)$. One iteration of speculative decoding works like this:

Draft. Run the draft model autoregressively for $k$ steps (say $k = 4$). At each step it samples a token from $q$ and feeds it back to itself. This produces a guessed continuation $g_1, g_2, \dots, g_k$ and the draft probabilities $q(g_i)$ at each position.

Verify in parallel. Feed the original context plus all $k$ guesses into the target model in a single forward pass. Because of the causal mask, this returns the target distribution $p$ at every one of the $k$ positions simultaneously: $p_1, p_2, \dots, p_k$, plus a bonus $p_{k+1}$ for the position after the last guess.

Accept or reject, left to right. For each guessed token $g_i$, accept it with probability $\min\left(1, \frac{p_i(g_i)}{q_i(g_i)}\right)$. If the target likes the token at least as much as the draft did, accept it outright. Otherwise accept it with reduced probability.

Stop at the first rejection. The moment a token is rejected, discard it and everything after it. Then sample one corrected token from the adjusted distribution $\text{norm}(\max(0, \, p_i - q_i))$ at that position.

Bonus token. If all $k$ guesses were accepted, you also get a free token sampled from $p_{k+1}$, the bonus position. So a fully-accepted round yields $k+1$ tokens from one target forward pass.

Each iteration emits somewhere between 1 token (everything rejected, plus the resample) and $k+1$ tokens (everything accepted, plus the bonus). The expected number of tokens per iteration is what drives the speedup.

Worked example: one round with k = 4 guesses

Context so far: "The capital of France is". The draft model guesses 4 tokens:

g = [" Paris", ",", " which", " has"]

The target model verifies all four in one pass and we walk left to right:

" Paris": target loves it ($p > q$), accept.
",": target is fine with it, accept.
" which": target assigned this low probability while the draft was confident ($p \ll q$). The acceptance ratio $p/q$ is small, the random draw fails, reject.
" has": discarded, because it came after a rejection.

We resample at the third position from $\max(0, p - q)$ and get " a". This round emitted 3 tokens (" Paris", ",", " a") from a single target forward pass. Next round starts from the new context.

Watching it run

The animation below runs the loop end to end. Both sides produce the identical 19-token output: the left does it the ordinary way, one target forward pass per token, while the right drafts $k = 4$ guesses with a draft model costing an eighth of a target pass and verifies them together. The two timelines share one clock.

Press Next step to advance one beat at a time, where each draft step and each verification is its own beat with a note on what just happened, or Play to run it straight through. Watch for the rounds where a guess is rejected (the round still commits the accepted prefix plus a resampled token) and the rounds where all four survive and earn the bonus token.

The counters under each timeline are the thing to watch. By the end the left has spent 19 target forward passes and the right has spent 5, plus 20 cheap draft steps and some wasted work on rejected guesses. Speculation does more total arithmetic; it just arranges that arithmetic into fewer serial steps, and serial steps are what you wait on.

Why It Stays Exact

The acceptance rule is not a heuristic. It is the same construction as rejection sampling, arranged so that the final token distribution equals the target $p$ exactly. This is worth seeing because it is the entire justification for using the technique at all.

Consider a single position. The draft proposes a token from $q$. The probability that the algorithm ultimately emits a specific token $x$ has two parts: it could be emitted as an accepted draft token, or as a resample after rejection.

Probability x is emitted $$P(x) = \underbrace{q(x)\min\!\left(1, \tfrac{p(x)}{q(x)}\right)}_{\text{accepted}} \;+\; \underbrace{P(\text{reject})\cdot\frac{\max(0,\,p(x)-q(x))}{\sum_{x'}\max(0,\,p(x')-q(x'))}}_{\text{resampled}}$$

The first term simplifies to $\min(q(x), p(x))$. The total rejection probability works out to $\sum_{x'} \max(0, p(x') - q(x'))$, which exactly cancels the normalizer in the resample term. Adding the two pieces:

The two terms collapse $$P(x) = \min(q(x), p(x)) + \max(0,\, p(x)-q(x)) = p(x)$$

The split $\min(q, p) + \max(0, p-q) = p$ holds for every $x$ regardless of what $q$ is. The draft model can be wrong, biased, or barely related to the target, and the output is still distributed as $p$. A bad draft model only makes you slower (more rejections), never less correct.

Intuition: The draft's job is to propose, not to decide. Where the draft over-claims a token's probability ($q > p$), the acceptance test trims it down. Where the draft under-claims ($q < p$), the resample step adds the missing mass back. The two corrections are exactly balanced so the result matches the target distribution, token for token.

Key limitation: Exactness holds only when the draft and target share the same tokenizer and the verification uses the identical sampling parameters (temperature, top-p) as the target would have. If you change temperature between draft and target, or use mismatched vocabularies, you break the guarantee. Most implementations require draft and target to be from the same model family for this reason.

Where the Draft Comes From

Everything above assumes you have a draft. The acceptance rate, the fraction of guessed tokens that survive, depends entirely on how well the draft predicts the target. This is where the variants differ.

Note: This section is a summary. For the full jungle, including ReDrafter, native multi-token prediction, suffix decoding, block drafters, and the arithmetic that decides which one wins on your workload, see The Speculative Decoding Zoo.

A smaller model from the same family

The original approach: pair a large target (say 70B) with a small draft (say 1B) trained on similar data. The draft is a real, independent model. This works well but has two costs: you must serve a second model (extra memory), and the draft's own autoregressive loop is still serial, just cheaper.

Self-speculation and early exit

Instead of a separate model, use the target model's own early layers as the draft. Medusa bolts several extra prediction heads onto the target so it predicts tokens $t+1, t+2, t+3$ all at once from the same hidden state. No second model, no separate weights to load, but the heads need training.

EAGLE: drafting in feature space

EAGLE is the current workhorse. Rather than predicting tokens directly, it trains a tiny autoregressive head that operates on the target model's feature vectors (the second-to-last layer activations), which are far more predictable than raw token IDs. It reuses the target's own LM head to turn features into tokens. EAGLE-2 and EAGLE-3 add a tree of candidate continuations verified together, pushing accepted tokens per pass to 4-5 on typical workloads.

n-gram and prompt lookup: no model at all

For tasks with heavy copying (summarization, code editing, RAG), the next tokens often appear verbatim in the prompt. Prompt lookup decoding skips the draft model entirely: it searches the existing context for a recent matching n-gram and proposes the tokens that followed it last time. Zero extra parameters, and on copy-heavy tasks the acceptance rate is high.

Method	Draft source	Extra memory	Needs training
Two-model	Separate small model	High (full draft model)	No (use existing model)
Medusa	Extra heads on target	Low	Yes (the heads)
EAGLE-2/3	Feature-space head + tree	Low	Yes (the head)
Prompt lookup	n-gram match in context	None	No

When It Pays Off

Speculative decoding trades extra compute (running the draft, and verifying tokens you may throw away) for fewer serial target passes. Whether that trade wins depends on three numbers: the acceptance rate $\alpha$, the number of guesses $k$, and the cost ratio $c$ between a draft step and a target step.

Expected tokens per round (greedy/uniform-α model) $$\mathbb{E}[\text{tokens}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}$$

Here $\alpha$ is the per-token acceptance probability and $k$ is the number of drafted tokens. With $\alpha = 0.8$ and $k = 4$, you emit about 3.4 tokens per target pass instead of 1, roughly a 3x reduction in expensive forward passes. The actual speedup is lower because the draft and verification add overhead, but 2-3x end-to-end is typical and well documented.

Key limitation: Speculative decoding helps most at low batch sizes, where decode is memory-bound and the GPU has spare compute. At high batch sizes, the GPU is already saturated doing useful work across many requests, so the "free" verification compute is no longer free, you are stealing it from other requests. In a heavily batched server, naive speculative decoding can reduce total throughput even as it improves single-request latency. Tune it for your operating point.

Intuition: The acceptance rate is everything. A draft that agrees with the target 80% of the time gives a big win; one that agrees 40% of the time barely helps and may hurt, because you keep paying for rejected tokens. This is why feature-space drafting (EAGLE) beat the two-model approach: predicting the target's own internal features is far easier than predicting its tokens from scratch.

The Rest of the Inference Toolkit

Speculative decoding attacks the serial-latency problem. It is one tool among several, and they compose. A production server uses most of these at once. Here is how the others fit, and what bottleneck each one removes.

Continuous batching

The single biggest throughput lever. Because decode is memory-bound, the marginal cost of adding another request to a batch is nearly zero, you load the weights once and run them against many requests' tokens together. Continuous (in-flight) batching goes further than static batching: instead of waiting for a whole batch to finish, it swaps a completed request out and a waiting request in at every decode step, keeping the GPU full. This is why vLLM and TensorRT-LLM exist.

PagedAttention and prefix caching

The KV cache is the memory hog that limits batch size. PagedAttention stores the cache in fixed-size blocks like OS virtual memory pages, eliminating the fragmentation waste of contiguous allocation and letting you fit more concurrent requests. Prefix caching shares cache blocks across requests that begin with the same tokens (a shared system prompt, few-shot examples), so common prefixes are computed once. Covered in depth in the KV cache post.

FlashAttention

Attention's intermediate $L \times L$ score matrix is huge and, written to HBM, dominates memory traffic during prefill. FlashAttention fuses the whole attention computation into one kernel that never materializes that matrix in global memory, tiling the work in fast on-chip SRAM instead. It does not change the math; it removes the memory round-trips. This is what makes long-context prefill tractable.

Quantization

Since decode is memory-bound, shrinking the weights directly shrinks the dominant cost. Weight quantization to INT8 or INT4 (GPTQ, AWQ) halves or quarters the bytes you move per token, with small quality loss. You can also quantize the KV cache to INT8 for the same reason. Quantization is the most direct attack on the memory wall, and it stacks cleanly with everything else.

Prefill-decode disaggregation

Prefill (processing the prompt) is compute-bound; decode (generating tokens) is memory-bound. Running them on the same GPU means each interferes with the other. Disaggregated serving runs prefill on one pool of GPUs and decode on another, sized and tuned independently, then ships the KV cache between them. See the DualPath post for how the cache transfer itself becomes the next bottleneck.

Technique	Bottleneck it removes	Helps latency	Helps throughput
Speculative decoding	Serial token dependency	Yes (large)	Only at low batch
Continuous batching	Idle GPU between requests	Modest	Yes (large)
PagedAttention	KV cache fragmentation	Indirect	Yes (more concurrency)
Prefix caching	Recomputing shared prompts	Yes (prefill)	Yes
FlashAttention	Attention memory traffic	Yes (prefill)	Yes
Quantization	Weight/cache memory bandwidth	Yes	Yes
Disaggregation	Prefill/decode interference	Yes	Yes (at scale)

Choosing What to Apply

These techniques are not alternatives; they are layers. A sensible order of adoption, by effort-to-payoff:

Continuous batching + PagedAttention. Non-negotiable baseline. Just use vLLM, SGLang, or TensorRT-LLM. Massive throughput gain, no quality cost, no tuning.

FlashAttention. Already built into those frameworks. Free win for long contexts.

Quantization. INT8 weights almost always; INT4 when memory is tight and you have validated quality on your tasks. 2-4x on the dominant cost.

Prefix caching. Turn it on if requests share system prompts or few-shot examples. Big win for chat and agent workloads.

Speculative decoding (EAGLE or prompt lookup). Add when single-request latency matters and your batch sizes are moderate. Prompt lookup first for copy-heavy tasks (it is nearly free); EAGLE when you can afford to train a draft head.

Disaggregated serving. Only at large scale, when prefill and decode genuinely contend and you have enough GPUs to pool them separately.

The unifying lesson: LLM decode is bottlenecked on memory bandwidth, not arithmetic. Every technique here is, at heart, a way to do more useful work per byte moved out of GPU memory, whether by batching requests, shrinking the weights, reusing cache, or verifying several future tokens in the pass you were going to pay for anyway.