Liquid AI released speculative decoding draft models for its LFM2.5 family, enabling inference speedups of up to 3.18x on H100 GPUs and up to 2.87x on edge devices like the M4 Max MacBook, without altering output quality.
The approach uses lightweight draft models to predict tokens and a target model to verify them in a single pass, reducing memory-bound latency.
Function-calling latency drops by 57% on average for LFM2.5-2.6B, and the models are available immediately in llama.cpp and SGLang with open-source integrations.
What happened
Liquid AI released draft model checkpoints for three LFM2.5 models—LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and LFM2.5-8B-A1B—using speculative decoding to speed up inference. The draft models, each around ~300M parameters, use a lightweight architecture to predict candidate tokens, which a target model then verifies in a single forward pass. Day-one support is available in llama.cpp and SGLang.
Why it matters
Speculative decoding trades minimal memory overhead for significant speed gains without changing output quality. On an H100 GPU, LFM2.5-8B-A1B achieves up to 3.18x throughput improvement, while on-device (M4 Max MacBook), LFM2.5-2.6B reaches 139 tokens per second on average—comparable to proprietary cloud models. For function-calling tasks, LFM2.5-2.6B's latency drops by 57% on average, making on-device agentic inference more practical.
What to watch
The draft model checkpoints are available now on Hugging Face in both Safetensors and GGUF formats (LFM2.5-2.6B-DSpark, LFM2.5-1.2B-Instruct-DSpark, and LFM2.5-8B-A1B-DSpark). Speedup varies by dataset and model size; LFM2.5-8B-A1B's on-device improvement is limited to 1.18x on average due to current MoE implementation constraints in llama.cpp's Metal backend.
Ask the AI about this article →
Speculative decoding has emerged as a practical technique to overcome the memory-bound bottleneck in LLM inference. The decode phase, where a model streams weights from DRAM to SRAM to produce each token sequentially, is the latency-critical component—not the computation itself. Liquid AI's DSpark approach combines three refinements: a parallel backbone (like DFlash) that produces hidden states for all draft tokens in one forward pass, a lightweight sequential Markov-chain head that captures inter-token dependencies and boosts acceptance rates at later positions, and a confidence-scheduled verifier that prunes low-confidence token sequences when verification would be more costly than single-token decode.
The draft models themselves are modest in size (~300M parameters across all three variants), trained on a diverse mix of SFT, chat, code, and function-calling data. Rather than optimizing for minimal loss, Liquid AI selected training epochs by acceptance rate—a direct proxy for inference speedup. This pragmatic choice reflects that the goal is not general accuracy but effective token-prediction fidelity under speculative decoding's constraints.
The empirical results show strong GPU performance: LFM2.5-8B-A1B hits 3.18x on MATH500 (428 → 1362 tok/s), while LFM2.5-2.6B averages 2.67x across five benchmarks on H100. On-device gains are more variable. LFM2.5-2.6B achieves 2.27x on average on M4 Max (61 → 139 tok/s), crossing the ~140 tok/s threshold of typical proprietary cloud offerings—a meaningful milestone for interactive on-device inference. However, LFM2.5-8B-A1B's on-device speedup drops to 1.18x average, constrained by the current implementation of mixture-of-experts (MoE) in llama.cpp's Metal backend; verifying multiple tokens activates more experts, increasing weight traffic relative to single-token decode. This gap illustrates that speculative decoding's gains depend on architecture and backend optimization, not just algorithmic correctness.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
