AIToday
Large Language ModelsAI Coding AssistantsAI Safety & AlignmentTHE DECODERPublished: Nov 23, 2026, 06:01 JST

Claude Opus 5.5, GPT-6 Astra finish under 1 in 10 tasks in sim

Send on LINE
Claude Opus 5.5, GPT-6 Astra finish under 1 in 10 tasks in sim

3 Key Points

  1. What happened

    In a simulated software company with colleagues, project boards and chat histories, Claude Opus 5.5 and GPT-6 Astra fully completed fewer than one in ten tasks, despite nearing top scores on benchmarks like SWE-bench.

  2. Why it matters

    Benchmarks may overstate how reliably these agents run a messy daily workflow on their own — they finish clear, standalone assignments quickly but lose track as autonomous developers.

WHO IT HITSEngineering managers and developer-tool buyers who gauge coding agents by benchmark scores may be overestimating how much autonomous work they can hand off in real team settings.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The experiment matters because it tests something the standard benchmark does not: continuity. Prior results, such as Anthropic's statement that Claude writes more than 80 percent of its production code, and the near-saturation of SWE-bench, paint a picture of agents that are highly capable in isolated, well-specified assignments.

The simulated-company setup keeps the messy parts in — colleagues, project boards and incomplete handoffs — and asks the agents to act as long-running developers. The gap between finishing clear tasks quickly and completing fewer than one in ten whole tasks suggests the weak point may sit in tracking work across a daily workflow rather than in producing code for a single prompt. That reading is an inference from the reported results, not a claim the body makes.

The headline names Claude Opus 5.5 and GPT-6 Astra because even the strongest models tested fell short, which is what makes the result notable against the background of benchmark saturation.

FAQ
What exactly did the agents have to do?
They worked as digital employees inside a simulated software company, with colleagues, project boards and chat histories, and had to keep developing software and close open tickets in a messy daily workflow.
How did the agents perform on simpler work?
They finished clear, standalone assignments quickly, but they lost track as autonomous developers over longer tasks.
What did Anthropic say about Claude's coding role?
Anthropic says Claude writes more than 80 percent of the company's production code.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleAnthropic capped banker teams at four, sought six-month rival bans