What is NVIDIA CUDA MPS?

CUDA Multi-Process Service is a binary-compatible alternative implementation of the CUDA API. It allows multiple processes to share a GPU concurrently without code changes, reducing context-switching overhead.

How does MPS differ from time-slicing and MIG?

Time-slicing gives each process exclusive GPU access sequentially, while MPS allows concurrent kernel execution from different processes. MIG creates hard physical partitions with dedicated memory controllers, whereas MPS uses a shared context with soft SM limits.

What are the prerequisites for deploying this solution?

You need an AWS account with access to EC2 g6e.4xlarge or g7e.4xlarge instances, NVIDIA drivers 535+ with CUDA 12.x, Docker with NVIDIA Container Toolkit, and the Triton Inference Server container. A Parakeet TDT 0.6B V2 model checkpoint is also required.

Audio & SpeechOpen-Source AIAmazon AI BlogAug 27, 2026Aug 28, 2026, 04:00 JST

NVIDIA MPS slashes ASR GPU needs by 75% on AWS

Send on LINE

3 Key Points

What happened
AWS, NVIDIA, and Heidi Health published a solution that cuts GPU infrastructure for automatic speech recognition (ASR) inference by 75 percent—from 16 GPU instances down to 4—using NVIDIA CUDA Multi-Process Service (MPS) with Triton Inference Server on Amazon EC2.
Why it matters
A single ASR request typically uses only 15–20 percent of a GPU's compute capacity, leaving 80 percent idle due to default time-slicing. By partitioning the GPU into concurrent instances, the setup maintains sub-second latency at 92.1 requests per second (RPS) per GPU, which could reduce operational costs for high-throughput speech services.
What to watch
The solution is implemented on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances (NVIDIA L40S, 48 GB) and requires NVIDIA drivers 535+ with CUDA 12.x. Code is available in an open-source repository, deployable with three commands.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The post describes how Heidi Health, an AI Care Partner processing over 2.4 million clinical consultations weekly across 190 countries, faced high GPU costs due to inefficient GPU utilization. Low per-request GPU usage and strict latency requirements forced the company to run 16 instances. The collaboration with AWS and NVIDIA produced a solution that leverages CUDA MPS to partition the GPU, achieving a 75% reduction in infrastructure while maintaining sub-second latency.

This optimization builds on an earlier post about fine-tuning the Parakeet TDT 0.6B V2 model. The current work focuses on serving that model efficiently. By combining MPS with Triton's dynamic and sequence batching, the solution addresses the specific bottlenecks of transcription and diarization workloads. The open-source repository offers a practical path for others facing similar challenges, potentially lowering the barrier to deploying ASR at scale on AWS.

FAQ

What is NVIDIA CUDA MPS?: CUDA Multi-Process Service is a binary-compatible alternative implementation of the CUDA API. It allows multiple processes to share a GPU concurrently without code changes, reducing context-switching overhead.
How does MPS differ from time-slicing and MIG?: Time-slicing gives each process exclusive GPU access sequentially, while MPS allows concurrent kernel execution from different processes. MIG creates hard physical partitions with dedicated memory controllers, whereas MPS uses a shared context with soft SM limits.
What are the prerequisites for deploying this solution?: You need an AWS account with access to EC2 g6e.4xlarge or g7e.4xlarge instances, NVIDIA drivers 535+ with CUDA 12.x, Docker with NVIDIA Container Toolkit, and the Triton Inference Server container. A Parakeet TDT 0.6B V2 model checkpoint is also required.

Amazon AI BlogRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.