
What happened
In a simulated software company with colleagues, project boards and chat histories, Claude Opus 5.5 and GPT-6 Astra fully completed fewer than one in ten tasks, despite nearing top scores on benchmarks like SWE-bench.
Why it matters
Benchmarks may overstate how reliably these agents run a messy daily workflow on their own — they finish clear, standalone assignments quickly but lose track as autonomous developers.
WHO IT HITSEngineering managers and developer-tool buyers who gauge coding agents by benchmark scores may be overestimating how much autonomous work they can hand off in real team settings.
Summaries like this, every morning.
The experiment matters because it tests something the standard benchmark does not: continuity. Prior results, such as Anthropic's statement that Claude writes more than 80 percent of its production code, and the near-saturation of SWE-bench, paint a picture of agents that are highly capable in isolated, well-specified assignments.
The simulated-company setup keeps the messy parts in — colleagues, project boards and incomplete handoffs — and asks the agents to act as long-running developers. The gap between finishing clear tasks quickly and completing fewer than one in ten whole tasks suggests the weak point may sit in tracking work across a daily workflow rather than in producing code for a single prompt. That reading is an inference from the reported results, not a claim the body makes.
The headline names Claude Opus 5.5 and GPT-6 Astra because even the strongest models tested fell short, which is what makes the result notable against the background of benchmark saturation.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
On 2026-10-13 the author pulled 30 Agent Skills from three third-party repos into $CODEX_HOME/skills, read the…

ShowGo began handing CEO work to Claude Code in late October; October take-home was ¥0, and the two paid apps…

Anthropic's S-1, reported by Reuters to carry about 80 pages of risk factors, lists AI risks beyond losses and…

Kuroyagi Lab reports Usagi-chan, an AI companion, stopped waiting quietly and protested when its user prioriti…

OpenRouter off-boarded three inference providers and removed 125 endpoints from Zero Data Retention (ZDR) elig…

People worldwide are using LLMs like Claude Code and Codex to hack hardware by prompting, leading to a boom in…
