Jakob Jordan, a nonspeaking young man with autism and apraxia, starred in an opera called Sensorium Ex using a custom AI speech synthesis device that makes his voice sound like him with full expressive control.
The device, built by NYU professor Luke DuBois, samples Jakob's own speech, extracts 401 parameters that define his voice's acoustic properties, and synthesizes new text through those parameters—letting him inject emotional affect (sarcasm, joy, sadness) into his delivery via hand-held lidar sensors that control speed, pitch, and timing.
The project flips the philosophy of speech synthesis from a disability aid that sounds like a computer to a communication tool that sounds like the person using it.
What happened
Jakob Jordan, a young man with autism and apraxia who cannot reliably speak aloud, starred in an opera called Sensorium Ex using a custom AI speech synthesis device designed by New York University professor Luke DuBois. The device samples Jakob's own disordered speech, infers 401 parameters that represent the acoustic architecture of his voice, and synthesizes new text through those parameters—allowing him to hear his thoughts spoken in a voice that sounds like him, with full expressive control via lidar sensors that let him speed up, slow down, and change pitch in real time.
Why it matters
Traditional text-to-speech devices for people with communication disabilities (AAC devices) treat the synthesized voice as generic—monotone and emotionless. Jakob's system shifts the frame: instead of just outputting text, it gives him what users identified as a civil rights need—the right to inject emotional affect, to sound sarcastic or joyful or upset. Composer Paola Prestini built the opera around this insight, working with poet Brenda Shaughnessy (who has lived experience as a mother to a nonverbal child) to honor the disabilities written into the characters and create a framework where disabled performers could thrive.
What to watch
The device runs entirely on a local computer using lidar sensors (the same $6 technology that deploys car airbags) rather than cloud services, preserving privacy by eliminating third-party data exposure. Jakob continues to use a traditional text-to-speech device for everyday life; the AI voice is his choice for artistic expression and performance. DuBois notes that entering what you want to say remains slow and cumbersome—the team's next challenge.
Ask the AI about this article →
The project emerged from composer Paola Prestini's decision to honor lived experience in her work: after an opera company commissioned her to create a piece about blindness, she insisted on collaborating with someone who had direct experience with the subject matter. She partnered with poet Brenda Shaughnessy, whose daughter is nonverbal and nonambulatory, and together they developed a libretto that asked questions neither could answer alone—questions that required technological innovation. This led Prestini to create not just an opera but an entire organizational framework (her company VisionIntoArt) to support disabled performers, and to recruit Luke DuBois and his team at NYU's Ability Project as technical partners.
DuBois's team began with design research, talking to existing AAC device users to understand their core frustrations. Users reported two main problems: the device does not sound like their own voice, and they have no agency in delivery—the speech synthesis plays back monotone and unchanging. The team's breakthrough was recognizing that this was not just a user experience issue but a communication civil rights issue: everyone deserves the right to sound sarcastic, joyful, upset, or hangry. The technology they built directly addresses this by letting the system learn from a user's actual voice (even disordered speech) and then give that user real-time expressive control. Notably, the system runs locally on standard hardware using $6 lidar sensors (the same technology that deploys car airbags), avoiding cloud services and privacy risks. In Jakob's words, the device felt like his voice was "released from prison after a wrongful conviction"—a shift from tools designed as aids administered to disabled people to tools designed to amplify and express what disabled people already want to say.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
AI giants are leaning into health care as a strategy to stall public backlash, according to Axios
Mitsubishi Electric and its U.S
Global funding to biotech startups has held between $36 billion and $40 billion for the past few years, and 20…
Researchers at Lille University Hospital in France rigged an LLM-based diagnostic support system to suggest a…
Social media conversations about Tempus AI intensified after Moderna and Merck reported Phase 3 success in per…
A Pew Research Center survey of 3,488 U.S
