Taste-Bench: GPT-5.6 Sol picks right path 59.7% of the time

The benchmark is built on a simple observation about how agents actually work: when the same task is attempted multiple times, the attempts eventually diverge, and the final results show which path was better. By freezing a run at that branching point and hiding all later work, the researchers turn ordinary agent runs into labeled comparisons. The raw data came from thousands of scored runs on coding and AI research tasks, including SWE-bench Pro and METR's MALT transcript release, and human review confirmed almost all of the auto-generated labels.

One pattern stands out in the results. When the key clue is already visible in the prior steps, average accuracy across all models reaches 62.3 percent. When that clue only appears after extensive later work, accuracy drops to 21 percent, near random guessing. The researchers repeated the test with three reasoning settings for two models, and switching from the lowest to the highest setting barely changed accuracy. Models spent the most reasoning tokens on the hardest tier, thinking longest on the questions they missed most often, which the team reads as a sign that the needed clue simply is not there yet at those forks.

The authors argue that this captures a different skill from classic end-to-end tests. Models that score similarly on SWE-bench Verified show clear gaps on Taste-Bench. They also show the skill can be taught: distilling a teacher model's judgment into a Qwen3.6-27B student raised accuracy on unseen tasks from 30 to 47.9 percent, and feeding that student's advice into a separate agent lifted SWE-bench Pro success from 14.6 to 33.7 percent, close to the 39 percent ceiling of perfect advice.