New reliability framework reveals that LLM agents performing well on short tasks fail dramatically on longer ones, challenging traditional AI benchmarking methods.

Existing pass@1 benchmarks measure single-attempt success but fail to capture reliability needed for production deployments across repeated long-horizon tasks

New framework introduces four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP)

Testing of 10 models across 23,392 episodes on 396 tasks shows reliability decay varies dramatically by domain—software engineering drops from 0.90 to 0.44 while document processing remains stable at 0.74-0.71

High variance amplification (VAF) emerges as a signature of capability tier rather than a flaw, suggesting performance degradation patterns differ fundamentally between model classes