Nerf Bench flags AI model declines after two readings outside 90%–110%

Nerf Bench is built around a single number called Power, and that number is deliberately broader than accuracy alone. A model that completes a task just as well but burns more tokens or costs more to run is recorded as having lost Power, which means a decline in the score does not necessarily mean answers got worse. The benchmark also sets its own baseline: the first test run at launch fixes the model's 100% reference point, and later runs are read against it.

The scoring rule is designed to avoid crying wolf. Drawing on the idea of variance in a normal distribution, Nerf Bench calls anything between 90% and 110% of launch performance ordinary fluctuation, and it only treats a change as real when results land below or above that range twice in a row. A drop confined to one or two tasks is not enough either; the benchmark looks for a broad pattern of decline before calling it performance degradation.

That caution extends to what the benchmark shows. Nerf Bench keeps its specific tasks, test contents and test results private, saying this is meant to protect the reliability of the results and curb benchmark gaming. It also publishes its methodology as a blog explainer, so the scoring logic is visible even though the underlying tests are not.