Deepgram's Nova-3 on AWS Trainium2 delivers 11–70 percent more audio per dollar

Deepgram's models process quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations, so serving efficiency directly determines how much audio the system can handle for a given infrastructure budget. High-volume batch workloads put heavy pressure on that budget, making inference efficiency a natural target. The porting effort began with Nova-3 as standard PyTorch code and moved to AWS Trainium2 through the AWS Neuron stack, with layer-level kernel replacements from the public NKI Library and a hand-written conv2d kernel that unlocked batch sizes above 32. A data-parallel Rust runtime removed Python and PyTorch from the serving path, reproducing reference token sequences exactly and improving throughput. The measured price-performance gain on a single trn2.3xlarge instance validates AWS Trainium as a viable option for Deepgram's inference workloads, and the benefits may extend to self-hosted deployments on Amazon SageMaker AI.