SGLang-Neuron now available via AWS Deep Learning Container

SGLang-Neuron arrives as an out-of-tree plugin, a design choice that keeps Trainium-specific code separate from SGLang's core. That separation is meant to let the two codebases iterate independently, so improvements on either side do not have to wait for the other. The plugin implements the Trainium-specific models, attention, mixture of experts (MoE), and graph runner, while SGLang core continues to provide the API server, request lifecycle, RadixAttention, and prefix-cache bookkeeping.

Underneath, TorchNeuron registers Trainium as a native PyTorch device. SGLang-Neuron uses standard PyTorch operations for the model execution path and adds kernels written with the Neuron Kernel Interface (NKI) for performance, compiling them into executable programs with torch.compile. For serving at scale, the plugin also supports PD disaggregation, where a prefill server computes the prompt KV cache and transfers it to a decode server over Elastic Fabric Adapter using NIXL.

The release is positioned for teams that already use SGLang's server interface and generation APIs: they can launch models and invoke endpoints the same way, with Trainium selected via the SGLANG_PLATFORM=neuron setting. A walkthrough uses a Trainium3 Amazon EC2 instance and the pytorch-inference-sglang-neuronx container, suggesting the path is aimed at practical deployment rather than a research preview.