Speech Recognition Gets 2.8x Faster With Simpler Decoder Change

Key takeaway

• Speech recognition systems waste most of their compute on processing silence rather than actual speech, making voice agents feel slow.

• A new decoder called Token-and-Duration Transducer fixes this by letting the model predict how many frames to skip instead of confirming each frame one at a time, achieving up to 2.82x faster decoding while maintaining accuracy.

• The change requires only one extra output head on the joint network, and NVIDIA's and Speechmatics's implementations already lead industry speed benchmarks.

3 Key Points

FAQ

How does Token-and-Duration Transducer actually work?

Instead of processing audio one frame at a time and always advancing by exactly 1 frame, TDT adds a second output head that predicts duration—how many frames to skip. If the model recognizes silence, it can jump forward 4 frames in one step instead of confirming "nothing" four separate times. Blanks must advance at least 1 frame; tokens can have duration=0 for fast speech where multiple tokens align to a single frame.

Does TDT hurt accuracy to gain speed?

No. On English speech recognition, TDT matches or beats standard RNN-T on word error rate (the standard accuracy metric) while decoding up to 2.82x faster. On speech translation (MUST-C V2), TDT picks up 1.26 BLEU points, a translation accuracy metric, over RNN-T while running 2.27x faster.

Who is using this in production?

Speechmatics, a real-time speech-to-text API used in production voice agents, implements this approach. Their API scored 1.07% pooled word error rate on the Pipecat voice agent benchmark across 1,000 samples.