Apple's DiscoSign brings discourse-level AI to sign language translation

Sign language processing systems have traditionally operated at the sentence level, ignoring discourse phenomena fundamental to sign language comprehension. This gap is what DiscoSign aims to close. The work, a collaboration involving Northeastern University and Gallaudet University, tackles three specific phenomena within a modular Large Language Model (LLM)-based translation framework: spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and concept-gloss consistency, ensuring stable mappings between English concepts and American Sign Language (ASL) signs.

Because traditional translation metrics fail to capture discourse-level quality, the authors also introduced a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by their framework. Experiments on sentence-level and discourse-level datasets showed that their approach significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality.

The stakes hinge on whether these discourse-level gains hold across different sign languages and real-world signing conditions, something the body does not address. For DHH users and developers of sign language generation tools, the framework's value will likely depend on how well the new evaluation metrics translate into practical, user-facing improvements.