Speculative decoding reduces the number of expensive sequential target-model steps by verifying several cheap token proposals at once. It helps most when proposals are accurate and inexpensive; under poor acceptance or heavy server load, the extra work can erase the gain.
Speculative decoding reduces sequential target-model work
Large language models generate text autoregressively: each new token depends on the tokens that came before it, so a conventional decoder performs a new target-model forward pass for every step. GPUs are powerful at parallel work, but this token-by-token dependency limits how much of that parallelism can be used during generation.
Speculative decoding changes the schedule rather than the target model’s intended behavior. A cheaper draft mechanism proposes several future tokens. The target model then verifies those candidates together. If multiple candidates are accepted, the system advances several token positions after one target verification round instead of one. The technique can therefore reduce sequential latency without simply replacing the target model with a smaller one.
The important caveat is that speculation introduces its own work. Drafting, verification, cache management and rejected proposals all consume resources. The deployment question is not whether speculative decoding can work; it is whether the accepted-token gain is large enough to outweigh that extra cost on the actual workload.
How draft-and-verify decoding works
In the classic arrangement, a smaller draft model predicts a short sequence of candidate tokens. The larger target model scores those positions in parallel and applies an acceptance rule. NVIDIA’s speculative-decoding guide describes this as a draft-target system in which the target accepts the longest valid prefix of proposed tokens and then continues generation from the last accepted position.
The verification step is what preserves the target model’s output semantics. The draft model is allowed to be wrong because its proposals are not trusted blindly. Rejection sampling compares draft and target distributions and rejects candidates that do not satisfy the acceptance rule. When a token is rejected, later candidates in that speculative branch cannot simply be kept as though the earlier token were valid.
This matters because speculative decoding is sometimes described too loosely as “letting a small model write several tokens for a large model.” The small model is not replacing the large one. It is proposing work that becomes useful only when the target confirms it.
The amount of useful progress per round therefore varies. A predictable continuation such as boilerplate, structured output or a common phrase may produce several accepted tokens. A difficult reasoning step, a rare identifier or a distribution shift can cause proposals to be rejected early.
Acceptance rate is the key variable, but not the only one
Acceptance rate measures how often draft tokens survive target verification. Higher acceptance generally means more useful tokens are obtained from each expensive target round. That is why acceptance rate appears so often in speculative-decoding benchmarks and tuning guides.
But acceptance rate alone is not enough to predict a speedup. A very accurate draft model may still be too expensive. If generating five proposals requires substantial compute or memory traffic, the system can spend much of the saved target time on the drafter. A cheaper drafter with slightly lower acceptance can sometimes deliver better end-to-end latency.
Proposal length matters for the same reason. Proposing more tokens creates more upside when the continuation is predictable, but it also gives the system more work to verify and more opportunities for later tokens to be discarded after an early rejection. The optimal proposal length can change with model pair, prompt type, sampling settings and current server load.
Cache pressure also enters the equation. A separate draft model usually needs its own state and may require its own KV-cache management. That overhead is small in some deployments and material in others, particularly when memory capacity is already tight.
| Speculation approach | Extra model cost | Operational complexity | Where it can fit |
|---|---|---|---|
| Separate draft model | Runs an additional smaller model | Model pairing, cache and scheduling | When a fast drafter aligns well with the target |
| Model-integrated drafting head | Lightweight extra head or feature path | Training or conversion support required | Stacks that support EAGLE-style or similar methods |
| Multi-token prediction head | Additional prediction heads inside the model | Architecture-specific | Models trained with native future-token prediction |
| Prompt lookup | No separate neural drafter | Depends on prompt repetition and lookup logic | Summarization, retrieval and repetitive contexts |
Different ways to speculate
The separate draft model is only one design. EAGLE-style systems use a lightweight drafting head attached to internal target-model features. NVIDIA describes EAGLE-3 as using fused features from multiple target-model layers to build candidate branches that are then verified by the target. The goal is similar to a draft model, but the proposal mechanism is more tightly coupled to the target representation.
Another family uses multi-token prediction heads trained to forecast several future positions. The proposals still need verification at inference time, but the model architecture contains the speculative capability directly rather than depending on an independently served drafter.
Prompt-lookup decoding is different again. Instead of asking another model to predict the future, it searches the existing prompt or context for matching token sequences and proposes an n-gram continuation. This can be especially attractive when the input already contains text likely to be reproduced, such as summarization, document extraction, code editing or templated transformations.
The SmartSpec research evaluates both draft-model-based speculation and prompt lookup. One practical advantage of prompt lookup is that proposing candidates can be much cheaper than running another neural model, so the technique can still help at acceptance rates that would be unattractive for an expensive drafter.
Why speculative decoding can lose under load
A lightly loaded server and a saturated server do not have the same optimization target. At low request rates, spare GPU capacity can be used for drafting and verification without displacing much other work. Under high load, speculative tokens compete with queued requests for compute, memory bandwidth and batching slots.
SmartSpec is useful here because it treats speculation as a scheduling decision rather than a fixed rule. The paper reports that aggressive fixed proposal lengths can degrade latency at high request rates, while a load-aware policy can reduce or disable speculation when the expected goodput no longer justifies the extra work. In other words, a technique that lowers single-request latency in a quiet benchmark can still reduce total useful work on a busy service.
This distinction also explains why headline speedups are difficult to generalize. A reported 2x or 3x result may depend on a particular target model, drafter, hardware platform, batch size, acceptance profile and arrival rate. None of those numbers should be assumed for a different production system without measurement.
The operational trade-off clarifies when speculative decoding negates target-model acceleration rather than improving it. For speculation to reduce latency, the time saved by accepting candidate tokens must exceed the cumulative cost of draft-model generation and target verification passes. In lightly loaded serving environments with low concurrency, the target model operates primarily under memory-bandwidth constraints, allowing speculative verification passes to run without delaying competing workloads. However, when request arrival rates rise and inference servers saturate compute capacity, batched tensor operations become compute-bound. In this saturated regime, evaluating speculative proposals across concurrent requests directly competes for GPU execution units and memory bus bandwidth. Consequently, when the acceptance rate falls below the break-even threshold, every discarded draft token adds scheduling latency and memory overhead without advancing the generation sequence, causing overall serving goodput to decline.
A practical deployment framework
1. Establish a clean baseline. Measure target-only time to first token, inter-token latency, throughput, tail latency, GPU utilization and memory use on representative traffic. Without that baseline, a speculative path can look faster in an isolated microbenchmark while hurting service-level metrics.
2. Measure proposal economics. Track average proposed tokens, accepted tokens, drafter latency and verification latency. Acceptance rate is useful, but the more actionable metric is the amount of target-model sequential work avoided per unit of extra compute.
3. Segment by workload. Chat, code completion, summarization and retrieval-heavy generation can have very different predictability. A single global proposal length or draft policy may be suboptimal. Workload-aware routing can be more effective than forcing speculation everywhere.
4. Test under realistic load. Run the same experiments at low, medium and near-saturation request rates. Watch for the point where extra drafting and verification begin to reduce goodput or increase tail latency.
5. Make speculation adaptive or reversible. The safest production design can reduce proposal length or turn speculation off when acceptance drops or the server becomes compute-bound. Keep the target-only path available as a fallback.
6. Re-test after model or prompt changes. A drafter tuned to one target model or traffic distribution may lose alignment after a model upgrade, tokenizer change, domain shift or new sampling configuration. Acceptance behavior should be treated as a monitored production metric, not a one-time benchmark.
Speculative decoding is most valuable when it removes expensive sequential target passes with cheap, high-quality proposals. The engineering work is in finding that operating region and staying inside it as workloads and server load change.
