NVRx on Amazon EKS keeps PyTorch FSDP training at 99%+ efficiency

The setup addresses a basic math problem in large-scale training: the longer a job runs across many nodes, the more likely some interruption will hit at least one worker. AWS frames the cost as twofold — idle GPU hours from faults, plus the wall time consumed by synchronous checkpointing, which it says ate up to 40% of total wall time at the cluster sizes tested.

The design splits the problem by fault class. Async checkpointing, through NVRx's TorchAsyncCheckpoint, hands the state dict to a background process so training continues; with FSDP's LOCAL_STATE_DICT, each rank writes its own shard with no all-gather. Recovery comes in two independent layers: in-process restart wraps the training function so transient exceptions or NCCL hangs don't kill the Python process, while the ft_launcher swaps in for torchrun and respawns workers after SIGKILL, OOM kills, or OS-level hangs. A cluster orchestrator would cover node loss. AWS notes the layers are independent and can be adopted separately.

The reported gains depend on the storage behaving as it did in these runs: because FSx Lustre write time was constant across 16 and 64 GPUs, async checkpointing could hide it entirely. Whether teams see similar numbers likely hinges on their own storage latency and fault patterns, not just the NVRx version they install.