KLダイバージェンス: 0.368 vs 0.511, why order matters

This is the sixth installment of Aoki's series on the mathematics behind AI and neuroscience, and it picks up from the previous episode's treatment of entropy. Entropy described the uncertainty of the world itself; KL divergence adds a second perspective, the extra information cost of believing a distribution that differs from reality. The worked example makes the abstract formula tangible: with natural logarithms the units are nats, and the two directions differ because the average is taken over whichever distribution is treated as real.

The piece then links the formula to a boxed identity, cross entropy equals entropy plus KL divergence, and to three properties: the value is never negative, it is zero only when the distributions match, and it is generally asymmetric. That last property is why KL divergence is not a distance in the usual mathematical sense. A further note points out that if the believed distribution assigns zero probability to something that can happen, the divergence becomes infinite, a warning about overconfident models.

Aoki extends the idea to continuous distributions, where the sum becomes an integral, and then to three application areas. In statistics, minimizing KL divergence is equivalent to minimizing cross entropy, which connects to maximum likelihood estimation from an earlier episode. In generative AI, the VAE objective includes a KL term that keeps the learned latent distribution close to a prior. In neuroscience, Karl Friston's free energy principle uses KL divergence to describe how the brain compares predicted and actual sensory information, though it is not established that the brain literally computes KL divergence.