GPT-6 Astra's weight edit lifts Qwen3.5-0.8B to 48.1% on MATH-500

Qwen3.5-0.8B's problem was not raw accuracy but a reasoning loop: on MATH-500, 93% of its answers ran into the 32,768-token ceiling before concluding. MIT CSAIL's Sparks of Weight-Space Programming addressed this by having GPT-6 Astra edit weights directly, and the result was a jump from 7.9% to 48.1% on MATH-500 with only 0.016% of weights changed. Astra worked under a 50-forward-pass constraint, trying combinations of layers and columns, but Nathan Barry's breakdown suggests most of that effort was really about finding which weights pushed along a single loop direction and by how much.

The mechanism runs through the residual stream, a 1,024-dimensional pathway in Qwen3.5-0.8B where each layer's output is added to the input rather than replacing it. Astra derived a loop direction by comparing residual streams from looping versus productive short texts, then edited the down_proj columns of each MLP that write into that stream. Because each token's push is linear in the per-column coefficients, the search becomes a linear program with an L1 penalty that naturally selects a sparse set of columns. Barry's estimate puts Astra's search at about 8 hours and 28 forward passes, while a PDHG GPU implementation reaches within 0.2% of the optimal value in about 1 minute.

The practical question is what gets lost in that speed-up. Fixing only 8 columns per layer came close to the original edit, and a heuristic direction auto-detected from loop traces still reached 47.8% on MATH-500. But it also flags open issues, including why 8 columns suffice and why selective constraints did not beat broad ones, pointing to a follow-up on PaPoo for those answers.