New ChartDiff benchmark with 8,541 annotated chart pairs reveals gaps in AI models' ability to compare charts and summarize their differences.

ChartDiff is the first large-scale benchmark specifically designed for cross-chart comparative summarization, addressing a major gap in existing chart understanding benchmarks that focus only on single-chart interpretation

The dataset contains 8,541 chart pairs from diverse sources with human-verified summaries describing differences in trends, fluctuations, and anomalies

Evaluation shows frontier general-purpose models achieve highest GPT-based quality scores, while specialized and pipeline-based methods score higher on ROUGE metrics but lower on human-aligned evaluations, indicating a mismatch between lexical overlap and actual summary quality