Hex cuts wrong edits 7x with evals on Quick Edits

Hex had already used evals for AI features, but Quick Edits was different: evals drove development from the beginning rather than serving as a final check. The feature runs on a small model that reads a request like "make the Enterprise line green" and proposes chart edits in one shot, handing off to the full Hex agent when the request is too complex. Hex started with a few hundred eval cases from real internal requests and grew that to 1,800, using multiple models to synthesize and fill gaps. Holdout cases made up almost 40% of the suite, and changes were accepted only when holdout results improved. Running 3 attempts per case was too noisy, with results differing by about 8 wrong edits, so Hex settled on 10 attempts per case, bringing holdout noise to 0.2 percentage points. Because a full run costs less than $10 and takes a few minutes, Hex reran the suite after every change, including three changes to chart properties in ten days. Notably, prompt changes alone rarely helped; fixes often meant moving logic into the validator or into property labels and descriptions the model reads.