How does the AI device actually work?

It samples Jakob's own disordered speech and uses an inference model to extract 401 parameters representing the acoustic architecture of his voice—the physical traits that make it sound like him (width of shoulders, depth of nasal cavity, tongue tightness, etc.). Normal written language is then synthesized through those parameters on a local computer, producing speech that sounds like Jakob but can speak any text.

What can Jakob control with the device?

Using lidar sensors attached to a platform or tray, Jakob can speed up, slow down, and change the pitch of his speech in real time as it plays back—allowing him to deliver words with emotional nuance and expressivity, like pausing between phrases or sounding sarcastic, instead of a flat monotone output.

Does Jakob use this device for everyday communication?

No. Jakob continues to use a more traditional text-to-speech device for daily life, including conversation. The AI-synthesized voice was developed specifically for the opera and the artistic control it enables.

Audio & SpeechAI in HealthcareHacker NewsAug 16, 2026Aug 16, 2026, 13:03 JST

AI gives nonspeaking opera singer expressive voice

Send on LINE

3 Key Points

What happened
Jakob Jordan, a young man with autism and apraxia who cannot reliably speak aloud, starred in an opera called Sensorium Ex using a custom AI speech synthesis device designed by New York University professor Luke DuBois. The device samples Jakob's own disordered speech, infers 401 parameters that represent the acoustic architecture of his voice, and synthesizes new text through those parameters—allowing him to hear his thoughts spoken in a voice that sounds like him, with full expressive control via lidar sensors that let him speed up, slow down, and change pitch in real time.
Why it matters
Traditional text-to-speech devices for people with communication disabilities (AAC devices) treat the synthesized voice as generic—monotone and emotionless. Jakob's system shifts the frame: instead of just outputting text, it gives him what users identified as a civil rights need—the right to inject emotional affect, to sound sarcastic or joyful or upset. Composer Paola Prestini built the opera around this insight, working with poet Brenda Shaughnessy (who has lived experience as a mother to a nonverbal child) to honor the disabilities written into the characters and create a framework where disabled performers could thrive.
What to watch
The device runs entirely on a local computer using lidar sensors (the same $6 technology that deploys car airbags) rather than cloud services, preserving privacy by eliminating third-party data exposure. Jakob continues to use a traditional text-to-speech device for everyday life; the AI voice is his choice for artistic expression and performance. DuBois notes that entering what you want to say remains slow and cumbersome—the team's next challenge.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The project emerged from composer Paola Prestini's decision to honor lived experience in her work: after an opera company commissioned her to create a piece about blindness, she insisted on collaborating with someone who had direct experience with the subject matter. She partnered with poet Brenda Shaughnessy, whose daughter is nonverbal and nonambulatory, and together they developed a libretto that asked questions neither could answer alone—questions that required technological innovation. This led Prestini to create not just an opera but an entire organizational framework (her company VisionIntoArt) to support disabled performers, and to recruit Luke DuBois and his team at NYU's Ability Project as technical partners.

DuBois's team began with design research, talking to existing AAC device users to understand their core frustrations. Users reported two main problems: the device does not sound like their own voice, and they have no agency in delivery—the speech synthesis plays back monotone and unchanging. The team's breakthrough was recognizing that this was not just a user experience issue but a communication civil rights issue: everyone deserves the right to sound sarcastic, joyful, upset, or hangry. The technology they built directly addresses this by letting the system learn from a user's actual voice (even disordered speech) and then give that user real-time expressive control. Notably, the system runs locally on standard hardware using $6 lidar sensors (the same technology that deploys car airbags), avoiding cloud services and privacy risks. In Jakob's words, the device felt like his voice was "released from prison after a wrongful conviction"—a shift from tools designed as aids administered to disabled people to tools designed to amplify and express what disabled people already want to say.

FAQ

How does the AI device actually work?: It samples Jakob's own disordered speech and uses an inference model to extract 401 parameters representing the acoustic architecture of his voice—the physical traits that make it sound like him (width of shoulders, depth of nasal cavity, tongue tightness, etc.). Normal written language is then synthesized through those parameters on a local computer, producing speech that sounds like Jakob but can speak any text.
What can Jakob control with the device?: Using lidar sensors attached to a platform or tray, Jakob can speed up, slow down, and change the pitch of his speech in real time as it plays back—allowing him to deliver words with emotional nuance and expressivity, like pausing between phrases or sounding sarcastic, instead of a flat monotone output.
Does Jakob use this device for everyday communication?: No. Jakob continues to use a more traditional text-to-speech device for daily life, including conversation. The AI-synthesized voice was developed specifically for the opera and the artistic control it enables.

Hacker NewsRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.