New AI safety monitoring system uses lightweight probes to efficiently screen LLM inputs while escalating complex cases to expensive experts with guaranteed cost controls.

Calibrate-Then-Delegate (CTD) introduces a novel Delegation Value (DV) probe that predicts whether escalating a case to an expert will actually improve safety decisions, rather than relying on uncertainty estimates.

The system enables probabilistic guarantees on computation costs and supports real-time streaming decisions by calibrating escalation thresholds using held-out data and multiple hypothesis testing.

CTD balances the competing demands of safety monitoring at scale by using cheap latent-space probes for routine screening while routing difficult cases to more expensive expert models only when beneficial.