OmniParse Bench: Datalab leads at 88.2, DeepSeek-OCR wins tables

The benchmark's design reflects a simple problem with most OCR comparisons: one number per parser tells you who is ahead on average, but not where a parser breaks. OmniParse Bench replaces that single score with thousands of pass/fail unit tests, each carrying a reason for failure such as "text missing" or "marked as table, not heading." The scoring runs as open code, and no LLM judges the output.

The page-level results show how differently parsers can fail. On the filled 1040 tax form, PaddleOCR-VL returned only the form's title and passed 0 of 8 tests, while Datalab passed 8 of 8 and DeepSeek-OCR passed 6 of 8. On handwritten notes, Datalab passed 5 of 6 tests while the other two passed 0. A single average folds a parse that returns only a title and one with a few wrong cells into the same number.

The benchmark builds on what olmOCR-bench showed, that pass/fail unit tests are a simple proxy for parsing quality anyone can check. Datalab's view is that olmOCR-bench has saturated, and what now separates top systems there is mostly output conventions and overfitting rather than reading. Each test also records how its answer was established, such as the publisher's own markup or agreement across a panel of independent models, so users can judge how solid a given test is.