Elon Litman: zero-loss PyTorch trick runs code in backward pass

Litman's starting point was a joke: could a loss function exist where loss.item() reads 0.0 but loss.backward() still moves the weights? The answer turned out to be yes, and far more consequential than a curiosity. The mechanism rests on a split between two views of the same expression. On the GPU, float32 above 2^30 can only represent multiples of 128, so adding a small value and subtracting 2^30 erases it. In PyTorch's autograd graph, the addition and subtraction nodes differentiate symbolically to 1 each, so the gradient flows through untouched.

Layering that with a stolen .detach() and a multiplication node that acts as a directional gate, Litman builds a scalar loss that is zero on paper over the reals, zero in float32 on the GPU, and non-zero only inside the backward tape. That is enough to set theta.grad to theta − F(theta), so one SGD step replaces the weights with F(theta). Wiring in the 3×3 neighborhood rule of Conway's Game of Life turns the parameter tensor into a board that advances one generation per optimizer.step(), while the logged loss stays pinned at 0.0.

The demonstration with a 1.6-million-parameter CNN on MNIST shows the payload and the real task can coexist: the auxiliary loss stayed at 0.0 across 2,345 steps, the network still hit 98.80% test accuracy against 98.84% for a baseline, and the 1,024 hidden-layer weights ran all 2,345 generations without a cell error. Litman suggests frontier models offer even more room, since unused vocabulary rows and unselected MoE experts receive an identically zero task gradient and could serve as an isolated scratchpad before writing a finished backdoor into an active layer.