AWS open-sources P-EAGLE to speed up AI inference

What happened

AWS invented Parallel-EAGLE (P-EAGLE) and contributed it to open source, a technique that predicts all speculative draft tokens simultaneously in a single forward pass instead of sequentially. The method is now natively supported in Amazon SageMaker JumpStart, available at launch for four models: GPT-OSS-120B, GPT-OSS-20B, Qwen3-Coder-30B-A3B-Instruct, and Gemma-4-31B-IT, with no manual drafter training or custom container setup required.

Why it matters

Speculative decoding (a strategy that uses a lightweight draft model to guess future tokens verified by a main AI model) has been limited by a hidden bottleneck: each draft token depended on the previous one, forcing sequential forward passes that grew slower with deeper speculation. By paralyzing the drafting phase, P-EAGLE eliminates this latency overhead, allowing enterprises to deploy faster AI inference endpoints without sacrificing accuracy or managing complex infrastructure.

What to watch

On real-world benchmarks running Qwen3-Coder-30B-A3B-Instruct on NVIDIA B200 GPUs with FP8 quantization, P-EAGLE delivers up to a 1.69x throughput speedup over vanilla EAGLE frameworks. Developers can deploy P-EAGLE-accelerated endpoints through SageMaker JumpStart with a single click or a few lines of code, with no need to manage underlying CUDA kernels or distributed serving setups.