What counts as a task being 'covered' by a benchmark?

Mercor treated a task as sufficiently covered when five or more test items from a benchmark were linked to it.

Which occupations were well covered versus poorly covered?

Software engineering, law, and accounting were relatively well covered, while truck driving and gas plant operation were rarely covered.

Why might some occupations be left out of benchmarks?

Mercor suggests occupations that are harder for current AI to handle, or harder for researchers to build tests for, may have been excluded.

Large Language ModelsAI Safety & AlignmentGIGAZINE AI2d agoOct 19, 2026, 22:00 JST

Mercor: AI benchmarks cover only 208 of 923 jobs

Send on LINE

3 Key Points

What happened
Mercor mapped 29 major AI benchmarks against O*NET's 923 occupations and found only 208 had at least part of their tasks covered by five or more test items.
Why it matters
This means benchmark results describe only a slice of the work AI could do, so claims that AI can substitute for labor may rest on thin evidence.

WHO IT HITSPolicymakers and business leaders who cite AI benchmark scores to justify automation or workforce plans should treat those scores as partial, since most occupations and physical-world tasks are not represented.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

FAQ

What counts as a task being 'covered' by a benchmark?: Mercor treated a task as sufficiently covered when five or more test items from a benchmark were linked to it.
Which occupations were well covered versus poorly covered?: Software engineering, law, and accounting were relatively well covered, while truck driving and gas plant operation were rarely covered.
Why might some occupations be left out of benchmarks?: Mercor suggests occupations that are harder for current AI to handle, or harder for researchers to build tests for, may have been excluded.

GIGAZINE AIRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.