What is NVIDIA CUDA MPS?

CUDA Multi-Process Service is a binary-compatible alternative implementation of the CUDA API. It allows multiple processes to share a GPU concurrently without code changes, reducing context-switching overhead.

How does MPS differ from time-slicing and MIG?

Time-slicing gives each process exclusive GPU access sequentially, while MPS allows concurrent kernel execution from different processes. MIG creates hard physical partitions with dedicated memory controllers, whereas MPS uses a shared context with soft SM limits.

What are the prerequisites for deploying this solution?

You need an AWS account with access to EC2 g6e.4xlarge or g7e.4xlarge instances, NVIDIA drivers 535+ with CUDA 12.x, Docker with NVIDIA Container Toolkit, and the Triton Inference Server container. A Parakeet TDT 0.6B V2 model checkpoint is also required.

Audio & SpeechOpen-Source AIAmazon AI Blog1d agoAug 28, 2026, 04:00 JST2 min read

NVIDIA MPS slashes ASR GPU needs by 75% on AWS

Send on LINE

Key takeaway

AWS and NVIDIA cut ASR GPU needs by 75% using MPS.
The change reduces instances from 16 to 4.
Latency stays under 650 ms at 92.1 RPS per GPU.

3 Key Points

What happened
AWS, NVIDIA, and Heidi Health published a solution that cuts GPU infrastructure for automatic speech recognition (ASR) inference by 75 percent—from 16 GPU instances down to 4—using NVIDIA CUDA Multi-Process Service (MPS) with Triton Inference Server on Amazon EC2.
Why it matters
A single ASR request typically uses only 15–20 percent of a GPU's compute capacity, leaving 80 percent idle due to default time-slicing. By partitioning the GPU into concurrent instances, the setup maintains sub-second latency at 92.1 requests per second (RPS) per GPU, which could reduce operational costs for high-throughput speech services.
What to watch
The solution is implemented on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances (NVIDIA L40S, 48 GB) and requires NVIDIA drivers 535+ with CUDA 12.x. Code is available in an open-source repository, deployable with three commands.

Ask the AI about this article →

Summaries like this, in your inbox every morning.

Get news like this every morning — free

Context & Analysis

The post describes how Heidi Health, an AI Care Partner processing over 2.4 million clinical consultations weekly across 190 countries, faced high GPU costs due to inefficient GPU utilization. Low per-request GPU usage and strict latency requirements forced the company to run 16 instances. The collaboration with AWS and NVIDIA produced a solution that leverages CUDA MPS to partition the GPU, achieving a 75% reduction in infrastructure while maintaining sub-second latency.

This optimization builds on an earlier post about fine-tuning the Parakeet TDT 0.6B V2 model. The current work focuses on serving that model efficiently. By combining MPS with Triton's dynamic and sequence batching, the solution addresses the specific bottlenecks of transcription and diarization workloads. The open-source repository offers a practical path for others facing similar challenges, potentially lowering the barrier to deploying ASR at scale on AWS.

FAQ

What is NVIDIA CUDA MPS?: CUDA Multi-Process Service is a binary-compatible alternative implementation of the CUDA API. It allows multiple processes to share a GPU concurrently without code changes, reducing context-switching overhead.
How does MPS differ from time-slicing and MIG?: Time-slicing gives each process exclusive GPU access sequentially, while MPS allows concurrent kernel execution from different processes. MIG creates hard physical partitions with dedicated memory controllers, whereas MPS uses a shared context with soft SM limits.
What are the prerequisites for deploying this solution?: You need an AWS account with access to EC2 g6e.4xlarge or g7e.4xlarge instances, NVIDIA drivers 535+ with CUDA 12.x, Docker with NVIDIA Container Toolkit, and the Triton Inference Server container. A Parakeet TDT 0.6B V2 model checkpoint is also required.

Amazon AI BlogRead Original Article

Send on LINE