SCIP shift tests: AI forecasters trail simple rules at 0.71 AUC

This test comes out of a real operational problem: when a shift-scheduling solve hits a time limit, someone has to decide whether to stop and return the current answer or grant more compute. Earlier work folded that decision into a mechanical stall detector — cut off if no new solution has appeared for a set period — without ever predicting whether an improvement was actually coming. This article asks the prediction question directly across 74 cases, with 24 of them improving and 15 of those improving by just one point.

The results cut against the stall intuition. Cases that had been quiet for 20–26 seconds clustered on the improving side of the scatter plot, because those were mid-size problems where the branch-and-bound tree was still working, not small problems near optimality or large ones stuck on root LP. The rule that follows the book's stall direction scored 0.37 AUC — worse than a coin flip — while the same signal reversed reached 0.63. Jev's cheaper, faster calls also came out below chance at 0.32 without framing; Sonnet 5 explained its reasoning but still read 12–18 node cases as low-probability when they in fact improved.

The practical read is that the size and direction of the stall threshold are local facts, not transferable heuristics: with only 36 cases to fit, even the best-performing logistic regression saw its threshold over-fire on holdout data and forfeit 35% of improvement points. Whether an AI judge earns a place in a scheduling pipeline therefore hinges on whether teams first collect labeled measurements from their own solver runs to validate the rule against, and on whether the extra 60 seconds is better spent on LNS rather than plain continuation.