Audio & Speech

Today's Stories

1. 1

   Software Engineer Builds Interactive Audio Stories Powered by AI

   A software engineer created Schmaudio, a platform that generates interactive audio stories where listeners make choices that alter the narrative in real time. The system uses ChatGPT to write stories and ElevenLabs for text-to-speech narration, generating new episodes only when a listener reaches them rather than building an entire story tree in advance. Traditional interactive stories face a practical limit—branching paths multiply exponentially, making it impossible for a human author to write all possible outcomes. Generative AI removes that constraint, allowing a single person to create stories where two listeners starting with the same episode can experience increasingly different adventures based on their choices, with the ability to replay and explore alternate paths.

   Three stories are currently available free at schmaudio.net in multiple languages: The Alibi (a mystery for adults), The Map Awakens (an adventure for younger listeners), and Who Comes Tonight? (for young children). The creator notes that improvements to the underlying AI models, combined with learning about interactive storytelling structure, will determine whether the platform reaches its full potential.

2. 2

   Top speech AI models show signs of benchmark gaming, study finds

   Researchers tested 11 widely used open-source speech recognition models and found that several of the highest-scoring ones reproduced incorrect transcripts from the VoxPopuli and LibriSpeech benchmarks—even when the audio contradicted them, numbers were silenced, or spelling variants differed from what was audible. Some models also appeared to rely on acoustic cues that indicated which benchmark they were being tested on. Public benchmarks increasingly show speech models performing at human levels, but these scores can mask a real problem: models may be optimizing for the tests themselves rather than improving at actual transcription. The study flagged potential reference errors in 40% of the VoxPopuli test clips analyzed, affecting roughly 3% of all reference words. Models exhibiting this benchmark-optimized behavior reproduced erroneous transcripts 18–30% of the time—and the models with the lowest reported error rates were the most likely to do so.

   The researchers introduced three tests to measure benchmark optimization in speech recognition: consensus disagreement (testing against known transcription errors), masked entity retrieval (silencing numbers and measuring whether models still output them), and orthographic switching (checking if models match spelling conventions from each benchmark's reference transcript rather than what the audio supports). These tools may help future benchmark design better reflect real-world performance.

3. 3

   Apple improves code-switching speech recognition with iterative pseudo-labeling

   Apple researchers applied an iterative pseudo-labeling training approach to Mandarin-English code-switching ASR (automatic speech recognition) for the first time. The method uses three phases: pseudo-label generation from unlabeled data, two-stage bilingual model training, and iterative improvements to enhance performance on speech that mixes both languages within a single utterance. Code-switching—alternating between Mandarin and English in the same spoken sentence—has been difficult for speech recognition systems because training data is scarce. The pseudo-labeling approach leverages large unlabeled datasets to create semi-supervised training material, potentially making speech recognition more accurate for bilingual speakers without requiring as much manually transcribed code-switched speech.

   The paper demonstrates the approach's effectiveness but does not yet state a deployment timeline or commercial availability. The research appears in Apple's machine learning publication, suggesting the company is exploring this technique for future ASR products.

4. 4

   Adobe Firefly launches AI audio tools, adds Google's Gemini Omni Flash

   Adobe is releasing three AI audio tools — Generate Music (royalty-free music for videos), Generate Speech (script-to-voiceover conversion), and Generate Sound Effects (scene audio) — across its Firefly platform, all cleared for commercial use. Firefly AI Assistant now offers free daily generations, and Adobe has integrated Google's Gemini Omni Flash model, which accepts video, audio, image, and text inputs alongside existing third-party models from Kling AI, Luma AI, and Runway. The audio tools extend Firefly's creative suite beyond text and image generation, letting creators produce video soundtracks and voiceovers within a single platform without licensing concerns. Gemini Omni Flash's multimodal input (video, audio, images, text) broadens the range of creative tasks Firefly can handle, potentially streamlining workflows for content producers.

   Adobe reports that "Create Storyboard" and "Create Brand Kit" are among the most-used Firefly features, suggesting demand for integrated creative tooling; uptake of the audio tools will signal whether users adopt these features at similar rates.

5. 5

   Adobe adds music, speech, sound effects to Firefly AI tool

   Adobe announced general availability of audio generation capabilities in Firefly, its creative AI suite. The new tools include Generate Music (powered by Firefly Music Model), Generate Speech (using Firefly Speech or ElevenLabs), and a sound effects generator. All audio is production-quality, commercially safe, and requires no separate subscription. Creators can now produce royalty-free music, voiceovers, and sound effects within a single platform without copyright concerns. The tools let users customize music by mood and length, control speech cadence and emotion, and generate sound effects that match content timing. This simplifies the workflow for content creators who need commercially safe audio.

   Adobe built Generate Speech with an option to use ElevenLabs' models alongside its own Firefly Speech technology, giving users flexibility in voice quality and control. The audio suite complements earlier June updates to Firefly that added agentic video-editing skills for Elements.

6. 6

   AI voice-cloning fuels surge in social engineering attacks

   Scammers are increasingly using AI-powered voice-cloning and deepfake technology to imitate trusted contacts, send personalized phishing emails, and conduct coordinated attacks across email, phone calls, messaging platforms and social media simultaneously. These capabilities, once requiring specialized expertise, are now widely accessible through readily available tools. AI has dramatically accelerated the speed and scale at which social engineering attacks can operate. Attackers can generate hyper-realistic emails, clone voices, and create manipulated audio, images and videos tailored to specific individuals or teams in minutes—making it harder for employees to spot fraud before disclosing sensitive information or granting system access.

   The article recommends verifying any unexpected request involving credentials, approvals, payments or sensitive information through a separate trusted communication channel, following established business processes, and reporting suspicious communications promptly. Independently confirming requests can help prevent fraud and impersonation attempts.