AssemblyAI cuts inference costs 70% with Amazon EC2 Spot

AssemblyAI's speech-to-text pipeline runs a chain of deep learning models on NVIDIA GPUs, and at scale that means thousands of GPU instances running continuously. Running entirely on On-Demand Instances would have been cost-prohibitive, so the company built its inference platform on Amazon ECS with capacity providers that mix Spot and On-Demand capacity. Spot Instances offer significant savings but can be reclaimed with a 2-minute warning, which is why AssemblyAI added a queue-based retry mechanism using Amazon SQS.

The architecture spreads demand across three GPU instance families—g4dn, g5, and g6—and multiple Availability Zones, drawing from a larger pool of Spot capacity. When an interruption occurs, an EventBridge-triggered Lambda function returns the in-flight job to the main SQS queue so another ECS task can process it. AssemblyAI also uses custom Amazon Machine Images containing model artifacts and container images to reduce recovery time to approximately 2 minutes. For end customers, the impact of an interruption is a small increase in processing time, but the transcription completes successfully.

The company runs fully independent copies of the architecture in US West (Oregon) and Europe (London) to support customers in regulated industries such as healthcare, finance, and legal, where audio data must stay within specific geographic boundaries. AssemblyAI's Speech-to-Text API is available on AWS Marketplace.