Hidden states, not behavior, catch doomed agents early in "Doomed from the Start"

The paper positions itself against a parallel line of work, AgentForesight, which instead reads observable trajectories and trains a 7B audit model to judge online whether a given step is a decisive error. The Doomed authors argue that behavior-only scorers are close to a coin flip in round 1 and only start working around rounds 3–4, by which point more than a third of all episodes have already finished, so the compute that could have been saved is already spent. Hidden states, by contrast, carry the outcome signal from the very first step, and adding behavior features to the hidden-state probe did not improve it further.

The engineering story is the recall-controlled gate cascade. Rather than a single threshold, gates sit at rounds 1 through 6, each calibrated with a Clopper–Pearson lower confidence bound and tuned jointly on validation data so the whole episode meets a target recall, then frozen and checked on independent data. That makes the trade-off a business dial: specify how many successful runs you are willing to sacrifice as a number, and the method maximizes the compute it can cut under that constraint.

The authors themselves list the caveats. Validation was effectively limited to a few environments and model scales, a shifted task distribution would break the exchangeability assumption behind the calibration, and the activations were extracted offline with teacher forcing, so production would require a separate serving-side implementation. Sparse calibration data caps the achievable recall at roughly 0.974 at their scale, and overall recall control is an empirically validated margin rather than a theorem.