AWS adds model caching to SageMaker HyperPod

Amazon SageMaker HyperPod is AWS's managed service for running large-scale machine learning workloads on Kubernetes. The cold start problem described in the post is a familiar pain for teams deploying large language models: before a pod can serve a single request, it must pull a multi-gigabyte container image and download model weights from storage, a process that can stretch to 30 minutes or more for models like DeepSeek-R1 at 600+ GB. This delay directly limits how quickly an autoscaling policy can respond to traffic spikes.

Model caching addresses the problem by moving both downloads off the critical path. The weights cache downloads model weights to local NVMe storage on each node before pods are scheduled, and the image cache pre-pulls the inference server container image. The two capabilities can be enabled together or separately, and both use preferred rather than required scheduling, so pods are never blocked if a cache is not ready. Benchmarks across models from 57–145 GB show around 60 percent faster scale-out with weights caching, and the image cache can remove over two minutes of cold image-pull time.

The trade-off is that caching is per-node, so storage consumption grows with the number of nodes, and the first population still requires a full download. Teams also need to pick an instance type with enough NVMe capacity for their model, and they must update the deployment spec to pick up new model weights, since source updates are not auto-detected. For organizations running self-hosted inference on HyperPod, the outcome likely hinges on whether their model sizes and instance types fit within the available NVMe storage, and whether the one-time cache population cost is acceptable for the faster scale-out they gain.