Discrete speech units struggle to preserve lexical tone information in Mandarin and Yorùbá despite SSL models encoding it naturally

Self-supervised learning (SSL) models successfully capture tone information in their latent representations, but quantization into discrete speech units (DSUs) loses this suprasegmental detail

Researchers tested tone languages Mandarin and Yorùbá and found DSUs prioritize phonetic structure over prosodic features like lexical tone

The limitation affects multimodal systems like text-to-speech and dialogue models that rely on DSUs for joint text-speech processing

Multiple quantization methods tested show the same problem, suggesting the issue is fundamental to how DSUs are derived rather than method-specific