NeoMME encoder: small, fast visual document retrieval

NeoMME is a response to the inefficiency of adapting generative visual language models for non-generative tasks like retrieval. By removing the causal decoder and separate vision tower, it cuts parameter and compute overhead while maintaining high performance. The 260M model's nDCG@10 of 0.523 on ViDoRe v3 is notable because it achieves this with under 300M parameters, outperforming models like ColQwen2.5 which is 3.75B but only slightly higher at 0.524. This suggests that for retrieval tasks, a well-trained encoder can rival much larger generative models.

The dual-head design (dense and late-interaction) provides flexibility for different use cases. The dense head is compact and supports fast ANN-based retrieval, while the late-interaction head preserves fine-grained local matches. The reported compression techniques (hierarchical token pooling and asymmetric quantization) substantially reduce index storage, making it feasible to deploy high-resolution retrieval at scale. The throughput advantage—encoding about 51 pages per second on an L40S—means lower GPU time and cost for indexing large document corpora, which is a practical concern for deployment.

NeoMME's release under Apache 2.0 and integration with Hugging Face Transformers lower the barrier for adoption. The model trains from scratch with a masked discrete-diffusion objective, which is a different approach from typical vision-language pretraining. While the body does not provide a direct comparison on all benchmarks, the Pareto frontier positioning and the storage/quality trade-offs indicate that this could be a practical choice for teams needing efficient multimodal retrieval without the overhead of a full VLM.