Analysis of open-weight LLM models reveals that BF16 format wastes about one-third of its bit budget on exponent storage, while sub-byte formats force models to adapt their weight distributions.

Researchers measured Shannon entropy across weight files from models by Qwen, DeepSeek, Google, OpenAI, Moonshot, MiniMax, NVIDIA, StepFun, and Zhipu—ranging from 0.6B to 1.4T parameters in formats including BF16, FP8, MXFP8, MXFP4, NVFP4, and INT4—and found that BF16 weights carry about 10.6 bits of entropy per element out of 16 allocated, with the slack concentrated in the exponent field (about 2.6 bits of entropy out of 8 allocated).

Weight magnitudes across all trained models cluster in a narrow band between 2^−7 and 2^−6, a regularity independent of architecture or training recipe. When distributions are normalized by mean and standard deviation, nearly every model collapses onto the same curve, showing that BF16's exponent is sized for a wider magnitude range than models actually use.

FP8 weights use roughly 80% of their 8-bit budget (about 6.5 bits of entropy), compared to BF16's 66%, by reducing mantissa precision rather than exponent slack. Below 8 bits, sub-byte formats (MXFP4, NVFP4) use per-block scales to factor the budget, forcing the weight distribution itself to narrow and adapt to the tighter constraint—the first format tier where the distribution changes rather than fits within a loose budget.