AWS SageMaker adds 100+ inference metrics to CloudWatch

What happened

Amazon SageMaker AI now sends more than 100 detailed metrics covering GPU health, token-level latency, KV cache pressure, traffic distribution across availability zones, and cold start diagnostics to CloudWatch. A new SageMaker Insights dashboard in CloudWatch displays these metrics across Performance, Capacity, and Reliability views, with automatic support for multi-model inference components. New endpoints have detailed observability enabled by default; existing endpoints require explicit opt-in.

Why it matters

ML platform engineers, MLOps teams, and site reliability engineers managing dozens of models and hundreds of GPU instances need to diagnose latency spikes and endpoint health issues in minutes. The shift from training to serving has made keeping inference endpoints healthy, responsive, and cost-efficient increasingly complex. These detailed metrics and a managed dashboard reduce reliance on custom Grafana and Prometheus setups.

What to watch

The metrics flow to CloudWatch within 2 minutes of an endpoint reaching InService status. Teams can also connect the metrics to third-party observability tools (Grafana, Datadog) through a PromQL-compatible endpoint. Detailed token-level metrics (like time to first token and inter-token latency) require vLLM or SGLang container frameworks.