How quantization shrinks large language models to run on consumer devices by reducing precision

A 7 billion parameter LLM trained in Float32 requires roughly 28 GB of VRAM to load; by switching to Float16, memory drops to roughly 14 GB, making it feasible for high-end consumer GPUs.

Float16 uses 16 bits split into 1 sign bit, 5 exponent bits, and 10 mantissa bits, losing 'fine detail' compared to Float32's 23-bit mantissa; BFloat16 preserves Float32's 8-bit exponent but reduces the mantissa to 7 bits, keeping range while sacrificing precision.

LLMs are resilient enough to function without exact weight values—knowing a weight is roughly 0.00046 instead of exactly 0.000458392 is often sufficient for correct operation.