Apple researchers propose CLaRa framework to compress documents for AI systems

Retrieval-augmented generation (RAG) enhances large language models with access to external knowledge by retrieving relevant documents for a given query, but the approach faces practical challenges: long document contexts slow down inference, and the retrieval and generation steps are often optimized separately, creating misalignment between which documents the retriever selects and which would actually help the generator produce better answers. Apple researchers—including Jie He, Richard He Bai, Sinead Williamson, Jeff Z. Pan, Navdeep Jaitly, and Yizhe Zhang—propose CLaRa (Continuous Latent Reasoning) to solve both problems. The framework compresses documents into dense vectors using embedding-based compression, which reduces the text length fed to the generator while preserving semantic information. To ensure compressed vectors remain retrievable and semantically rich, the team introduces SCP (a key-preserving data synthesis framework based on question-answering and paraphrase supervision), which generates training data that teaches the system how to compress while retaining answers and meaning. CLaRa then trains the reranker (which selects relevant documents) and the generator (which produces answers) end-to-end via a single language modeling loss, with gradients flowing through both modules using a differentiable top-k estimator. This unified optimization theoretically aligns retrieval relevance with answer quality—meaning the retriever learns to pick documents that actually help generate correct answers, not just documents that seem semantically similar to the query. Experiments across multiple question-answering benchmarks demonstrate that CLaRa achieves state-of-the-art compression and reranking performance, even at a text compression rate of 16, outperforming text-based fine-tuned baselines. The work was accepted at the UncertaiNLP workshop at EACL 2024.

Retrieval-augmented generation has become a standard technique for enhancing language models with external knowledge, but the approach carries two significant limitations: the need to process long document contexts slows inference, and the retrieval step and generation step are typically optimized separately rather than as a unified system. Apple's CLaRa framework addresses both constraints by compressing documents into dense vectors before they reach the language model, thereby reducing computational overhead, while also training retrieval and generation components end-to-end with shared gradients. The key innovation is the SCP data synthesis method, which creates compressed vectors that remain semantically rich enough to support high-quality question-answering—a balance that prior approaches struggled to maintain at high compression rates. By demonstrating state-of-the-art results even at a compression rate of 16, CLaRa suggests that the unified optimization approach can recover performance while significantly shrinking the effective context length, which is relevant for practitioners deploying models on-device or in resource-constrained environments where inference speed and memory footprint are critical.

No comments yet. Be the first to share your thoughts!