REVERSAL-BENCH shows the reset-free RL cliff

The premise behind autonomous reinforcement learning is continuous policy training without external resets — an agent that keeps learning on its own. That premise has quietly rested on an assumption: that the environment is reversible, meaning mistakes can be undone. In real-world manipulation, they often cannot. Pushing an object off a table or spilling a granular substance is not something an agent can reverse.

REVERSAL-BENCH was built to test that assumption directly. By controlling reversibility through a continuous parameter ρ∈ [0, 1], and by adding a reset oracle to verify state recoverability, the authors could dial irreversibility up and watch what happened across eight manipulation settings in five physics engines. The result was a sharp reversibility cliff: reset-free agents were consistently absorbed into irrecoverable states as ρ increased, while episodic agents maintained steady learning. The failure showed up across autonomous reset-free baselines and constrained RL, and persisted in full physics simulations under learned manipulation policies.

The authors also ran a control that matters for interpretation. By comparing against geometrically identical reversible counterparts, they confirmed the breakdown was causally driven by irreversibility rather than obstacle complexity — the same layout, made reversible, did not produce the same failure. A safety shield that intervenes before irreversible failures could predict recoverability accurately, but active recovery mostly worked only when the agent could physically steer clear of the trap. The stakes therefore hinge on whether recovery is physically available at all, not just predictable.