AIToday
Large Language ModelsAI Safety & AlignmentAI WeeklyPublished: Nov 21, 2026, 10:00 JST

METR Time Horizon 1.1: frontier models flunk eval design

Send on LINE
METR Time Horizon 1.1: frontier models flunk eval design

3 Key Points

  1. What happened

    METR released Time Horizon 1.1, expanding its benchmark from 170 to 228 tasks and scoring judgment-heavy work for the first time. No model produced an AI evaluation METR employees considered useful; Opus 5.5 reached only "borderline above the bar."

  2. Why it matters

    Models still double their task time horizon roughly every seven months on coding and math, but they fall below human performance on judgment tasks, suggesting "research taste" could be a major bottleneck for automated AI research.

WHO IT HITSAI research teams and benchmark designers who rely on METR's time-horizon charts to forecast progress now face a clearer split: strong coding and math gains, but weak judgment that may limit automation of research itself.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

METR's update arrives as frontier agents keep showing a consistent shape: strong on bounded technical tasks, brittle on open-ended ones. The doubling of time horizons every 3.6 months on verifiable technical work stands in contrast to the judgment side, where models landed below human performance on scoring philosophical arguments — though METR staff themselves disagree about how relevant that test is to automating AI research.

The benchmark's expansion to 228 tasks and its first formal scoring of judgment-heavy work mark a shift in what is being measured, not just how well models perform. Computer scientist Sayash Kapoor framed the stakes, arguing that "research taste" — choosing promising questions, hypotheses and experiments — "could be a major bottleneck" on the path to automated AI research. METR's own caution that results above 16 hours remain unreliable suggests the plot's most eye-catching numbers sit inside the error bars rather than above them, a reliability issue that compounds as task lengths grow.

FAQ
What is METR's Time Horizon 1.1?
It is an updated benchmark that expanded from 170 to 228 tasks and, for the first time, scores models on judgment-heavy work such as designing AI evaluations and scoring philosophical arguments.
How did Opus 5.5 perform on the eval-design task?
Opus 5.5 reached what METR calls "borderline above the bar" — not yet usable for its research, but promising.
Why does METR caution about its own results above 16 hours?
METR's Matt Putz acknowledged that human completion times at extended task lengths are now estimated rather than measured, reducing benchmark reliability at the frontier.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleGoogle.com drops search button in AI homepage redesign