AWS now allows language model providers to split inference into specialized phases on separate GPUs—one handling prompt processing, the other token generation—connected via fast networking. This eliminates delays when multiple users request answers simultaneously, particularly for long-context queries like document analysis or retrieval-augmented search. The feature is available today on SageMaker HyperPod with the latest Inference Operator.
Summaries like this, in your inbox every morning.
Sign up free →What happened
AWS SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), a technique that splits language model inference into two phases on separate GPU pools. Long prompts are processed on prefill GPUs, then passed to decode GPUs for token generation, eliminating the latency spikes that occur when both run on the same hardware.
Why it matters
Organizations serving concurrent users with long-context requests—such as chat assistants, document analysis, or retrieval-augmented search—experience stalls when a single long prompt blocks token generation for other requests. DPD removes this bottleneck by design, delivering consistent per-token latency under concurrency. This is especially valuable for streaming workloads where latency matters to the end user.
What to watch
DPD requires HyperPod Inference Operator version 3.2 or later and is deployed across separate prefill and decoder nodes with EFA RDMA networking. It works best for workloads with input prompts regularly exceeding 4,096 tokens and multiple concurrent users; shorter-prompt, lower-concurrency deployments may not benefit enough to justify the cross-GPU KV cache transfer cost.
Large language model inference has always posed a challenge at scale because the two phases of computation—prompt processing (prefill) and token generation (decode)—have fundamentally different performance characteristics. Prefill is compute-bound and can process entire prompts in parallel, while decode is memory-bound and generates one token at a time. When both run on the same GPU, long prompts monopolize resources and stall token generation for other concurrent requests, creating the per-token latency spikes that frustrate end users in streaming applications.
AWS's DPD implementation solves this by disaggregating the two phases across separate GPU pools connected via Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA). The system uses an intelligent router to decide on a per-request basis whether to send short prompts directly to a decoder (avoiding the overhead of cross-GPU transfer) or route long prompts through a prefiller first. The prefiller computes the key-value cache—the context the model needs to generate each token—and pushes it to the decoder over EFA, where full token generation begins. Because decoders never execute prefill, their latency remains stable even as concurrent requests arrive, and adding a long-context request no longer disrupts tokens already streaming.
The architecture is built on the vLLM production stack router and includes an optimization layer (LMCache) that caches recurring prefixes in CPU memory. When system prompts or retrieval contexts repeat, they can be served from CPU without GPU recomputation, speeding time to first token. AWS ships the networking stack pre-integrated on HyperPod, so the deployment is straightforward for workloads that meet the threshold—primarily high-concurrency services with prompts exceeding 4,096 tokens. For simpler, lower-concurrency, or short-prompt-only traffic, the fixed cost of KV cache transfer over the network outweighs the benefit of disaggregation, and traditional colocated inference remains the better choice.
No comments yet. Be the first to share your thoughts!
Log in to join the discussion
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
