AWS ships Ray LLM container for multi-node inference on Amazon EKS

The container builds on an earlier AWS post that introduced the Ray Serve DLC, which stood up a small model behind an HTTP endpoint on a single GPU node. The new release extends that approach to distributed serving, where customers previously had to assemble the dependency chain themselves: choosing compatible libraries for the GPU stack and debugging issues when a component drifted out of alignment. AWS describes that as undifferentiated work that takes time away from delivering models.

Under the hood, the deployment separates prefill, which processes the input prompt in parallel and builds the KV cache, from decode, which generates the response one token at a time. Prefill is compute bound and decode is bound by memory bandwidth, so sharing one GPU lets the bursty prefill work interfere with the steady decode work. Putting each phase on its own node lets them scale independently — but it also means the KV cache must be handed between nodes, which is why the high-bandwidth EFA interconnect is part of the package.

The container is positioned as one of several multi-node patterns, alongside tensor and pipeline parallelism for models too large for one instance, and data parallelism for throughput. From the described setup, users could add decode replicas for more throughput or move to larger models on bigger GPU instances, though the body does not state what AWS plans next.