Researchers identify token gradient cancellation as critical mechanism to stabilize reinforcement learning training in reasoning models

Paper addresses training instability issues in sparse reward reinforcement learning, including learning tax accumulation, solution probability drift, and entropy collapse

Proposes that intra-group comparison objectives must maintain gradient exchangeability across tokens to prevent reward-irrelevant drift during long-term training

Identifies two common mechanisms that disrupt exchangeability and prevent natural gradient cancellation on weak-credit, high-frequency tokens

Introduces minimal intra-group transformations to restore gradient cancellation structure, showing improvements in training stability and sample efficiency in experiments