Speculative Decoding Goes Mainstream: 1.5–2.5× Speed Without Quality Loss

Speculative decoding is a mathematically proven speedup technique that has circulated in research and power-user circles for years but is now becoming a standard feature in shipping models. The core idea is simple: instead of generating tokens one at a time (a sequential process), a small draft model guesses the next several tokens, and the main model then checks all of those guesses in a single parallel pass. If the main model agrees with the draft, that token is accepted instantly; if it disagrees, that guess is discarded and the main model's choice is used instead. The key insight is that this is lossless. The verification step uses a rejection-sampling rule that provably reproduces the main model's output distribution exactly, so the user gets identical output to what the large model alone would have produced—no quality tradeoff, only speed.

Research papers have documented speedups ranging from 2 to 3× on datacenter hardware. The original 2022 Google paper (Leviathan et al.) reported 2–3× on T5X; DeepMind's 2023 speculative sampling paper (Chen et al.) achieved 2–2.5× on a 70B Chinchilla model; Medusa (2024) reported 2.2–3.6× using extra decoding heads on the model itself; and EAGLE (2024) achieved 2.7–3.5× on Llama 2 70B. Local speedups are typically smaller; users report 1.5–2.5× as a realistic outcome on consumer hardware.

The traditional setup required manual pairing: finding a compatible small draft model (same family—Qwen drafts for Qwen, Llama for Llama—and roughly one-tenth the main model's size). LM Studio's guidance was a 1B draft for an 8B model, up to 1.5B for a 32B model. The user would specify both models and monitor the acceptance rate (the fraction of drafted tokens the main model agreed with). An acceptance rate of 70–80% roughly doubled speed; below 60%, the overhead became a net loss.

What changed in 2026 was the introduction of multi-token prediction (MTP) layers trained directly into the model. DeepSeek-V3's technical report in late 2024 mainstreamed the idea, noting that an extra head trained with an MTP objective "can also be reused for speculative decoding." In 2026, major model makers shipped with built-in drafters: Qwen 3.6 ships MTP layers, and llama.cpp added support this spring with the --spec-type mtp flag. One user running Qwen 3.6 27B on an M2 Max 96GB reported "2.5x speed increase, bringing it to 28 tok/s." Google's Gemma 4 went further, shipping official MTP drafter checkpoints for every model size in May 2026, including a 78-million-parameter drafter for the E2B phone-sized variant. Support landed across Ollama (via a DRAFT command in the Modelfile), vLLM, SGLang, and MLX. Google claims up to 3× speedup "while guaranteeing the exact same quality as standard generation."

The speedup works best in specific scenarios. Dense models running slowly (8–30 tokens per second) on deterministic work—code generation, information extraction, structured output, low-temperature generation—see the most gain. N-gram speculation (which drafts by pattern-matching recent tokens with zero extra VRAM cost) offers a free alternative for tasks like code editing, retrieval-augmented generation, and summarization. But the technique backfires consistently in three cases. Fast Mixture-of-Experts models already doing 100+ tokens per second see little or no improvement; an independent spring 2026 test on an RTX 3090 running Qwen3.6-35B-A3B (a fast MoE) showed a baseline of 135.7 tokens per second, a best speculative result of 131.1 (3% slower), and worst-case results 15% slower, even with acceptance rates near 100%. The problem is that each drafted token can pull a different expert slice into compute, so verifying a batch loads far more weights than generating one token, eliminating the parallel-verification advantage. Creative writing at high temperature also fails: acceptance rates collapse when there is no single predictable continuation, making the draft overhead lose more than it gains. Finally, VRAM pressure matters: if adding a 1B draft forces the main model to drop to a lower quantization or shrink the usable context window, the user has traded away more than they gained.

For 2026, the decision is straightforward. If running Qwen 3.6 or Gemma 4, enable the built-in MTP drafter—zero pairing work, matched by design. For dense models feeling slow (8–30 tokens per second) on coding or structured tasks, turn it on and watch the acceptance rate; above roughly 70%, you are winning; below 60%, change the draft or disable it. For any model, try n-gram speculation first if memory is tight. Skip it entirely on fast MoE models already doing 100+ tokens per second. Measure tokens per second before and after on your own workload and hardware, because as the RTX 3090 results show, the only benchmark that counts is your own machine.

Speculative decoding addresses a fundamental bottleneck in local large-language-model inference: token generation is memory-bandwidth-limited, not compute-limited. A GPU that processes an entire prompt at 800+ tokens per second grinds to 20 tokens per second during generation because each new token requires loading the model's weights from VRAM, and the arithmetic units sit idle waiting for data. The technique converts generation into a parallel verification task by having a small draft model guess several tokens ahead, then checking all guesses simultaneously on the main model—a kind of work GPUs excel at.

What changed in 2026 is the source of the draft model. Previously, users had to manually pair a compatible small model (e.g., Llama drafts for Llama, same family required) and tune the size ratio (roughly one-tenth the main model's parameters, or smaller). Now model makers embed multi-token prediction heads directly into the model during training, as DeepSeek-V3 did in late 2024 and Qwen and Google formalized in 2026. This eliminates setup friction: the drafter is perfectly matched, ships in the box, and requires only a command-line flag to enable. Ollama, llama.cpp, and LM Studio all added support for the new format.

The gains are real but conditional. Dense models crawling at 8–30 tokens per second on deterministic work (code, extraction, structured output) benefit most, often seeing 1.5–2.5× speedup with zero quality loss—the verification step mathematically reproduces the main model's output. But fast Mixture-of-Experts models (already 100+ tokens per second) see little or no gain because their compute is not idle; each drafted token can pull a different expert slice into memory, so batched verification loads more weights than single-token generation. Creative writing at high temperature also suffers, since the draft model's guesses become too unpredictable. The acceptance rate—the fraction of guesses the main model agrees with—is the diagnostic: above roughly 70% wins speed; below 60% costs more than it saves.

No comments yet. Be the first to share your thoughts!