AWS adds CreateAIBenchmarkJob sweeps to size SageMaker AI endpoints

Right-sizing a generative AI endpoint has traditionally meant deploying a model, running load tests by hand, adjusting capacity, and repeating that loop until the numbers look acceptable. The failure modes are costly in both directions: too many instances waste budget on idle GPUs, while too few cause requests to queue and latency to spike. AWS's post frames concurrency sweeps as a way to replace that trial and error with a systematic benchmark that raises simultaneous traffic in controlled steps.

The walkthrough itself uses NVIDIA Nemotron-3 Nano 30B, a Mixture-of-Experts model with 3B active parameters, deployed on an ml.g7e.2xlarge instance backed by an NVIDIA Blackwell GPU. AWS describes the settings used to serve it, including a flag required by the model's Mamba-Transformer hybrid architecture and a GPU memory utilization of 0.85 to leave headroom for cache growth under load. The benchmark then simulates traffic shaped like RAG or summarization work — 1,024 input tokens and 256 output tokens on average — and writes per-level metrics to Amazon S3.

The stakes hinge on whether teams treat the sweep output as a planning input rather than a one-off test. The post also offers an automated search recipe that finds the highest concurrency meeting one or more SLA thresholds, which AWS says can converge in fewer iterations than a linear sweep. Whether that holds across different models and instance types is likely to determine how much manual benchmarking work teams can retire.