AIToday
Large Language ModelsZenn AI/MLPublished: Nov 21, 2026, 04:00 JST

Claude Opus 5.5 hits 50.79% on ARC with no gradient descent

Send on LINE
Claude Opus 5.5 hits 50.79% on ARC with no gradient descent

3 Key Points

  1. What happened

    Running the official harness, Claude Opus 5.5 wrote an ARC model with no gradient descent and no forward calls, scoring 50.79% on the 3,530-question test set, against the paper's 50.8%.

  2. Why it matters

    The result reproduces the paper's figure almost exactly, suggesting the approach is repeatable rather than a one-off, and that Opus spent its roughly 45 minutes and 245,034 output tokens building a knowledge base and search procedure rather than training weights.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The paper, Weight-Space Programming Doesn't Need Transformers (arXiv 2611.22055) by POSTECH's Hyunjong Ok and Jaeho Lee, asks frontier LLMs to write a model's weights directly rather than train them by gradient descent. Under the harness rules, the writer sees only 10 ARC examples (five ARC-Easy, five ARC-Challenge) and must deliver vocab.txt, build.py, arch.py, and NOTES.md; arch.py must be a fixed tensor program with no if/while branching and no Python-side value extraction. The reproduction's 50.79% sits within a hair of the paper's 50.8% for Claude Opus 5.5's first-generation free-architected model, and re-scoring the repository's bundled model with harness.score_model gave 50.80%, matching the stored result across all items.

Beyond the headline number, the write-up records practical friction: three harness failures tied to tooling (a venv under /tmp hidden by a tmpfs overlay, a Unix socket path exceeding the roughly 108-byte Linux limit, and a venv under the home directory tripping the self-test), resolved by placing venv, data, and Hugging Face cache all under /opt/wsp. A separate comparison shows how much the result depends on scale of effort: a smaller model written outside the harness, with 155 fact sentences and 185,641 weights (3,922 non-zero), reached 31.33% on the test set and 28.51% on validation, versus 50.79% and 51.10% for the harnessed Opus version. Since both had zero forward calls, the gap appears to reflect the amount of written knowledge and the refinement of the retrieval procedure rather than extra attempts — though that reading is the author's interpretation.

FAQ
What exactly is the model the LLM writes?
According to the writer's NOTES.md, Opus built a hand-written science knowledge base of 4,071 lines searched by a fixed procedure, with 995,784 weights of which 714,879 are non-zero.
Why did the run stop before finishing?
The writer hit Claude's 5-hour usage limit mid-improvement; the harness scored the model at that point and marked the result valid.
How did the harnessed version compare to one written outside the harness?
The version written outside the harness, built from 155 fact sentences with idf-weighted bag-of-words and cosine similarity, scored 31.33% on the ARC test set.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleUS tech giants shift AI race to agentic action, led by Amazon and Google