New benchmark tests whether large language models can understand population-level trends and patterns across multiple texts, not just find isolated facts.

Text2DistBench is a new reading comprehension benchmark designed to evaluate LLMs' ability to infer distributional knowledge from collections of natural language text

The benchmark uses real-world YouTube comments about movies and music, requiring models to estimate proportions (like positive vs. negative sentiment) and identify frequent topics among viewers

Unlike traditional reading comprehension tasks that focus on factual information localized in specific passages, this benchmark emphasizes understanding population-level trends and preferences

The benchmark features a fully automated construction pipeline that is continuously updated, enabling reliable and long-term evaluation of LLM capabilities