Liquid AI has released 4-bit compressed versions of its LFM2.5 language models that retain 97% of full-precision accuracy, allowing developers to run them efficiently on edge devices like phones, laptops, and Raspberry Pi without the usual quality loss from compression.
The models were trained using Quantization-Aware Distillation, which teaches a smaller quantized model to mimic a high-precision teacher, and are available now on Hugging Face.
What happened
Liquid AI released QAD Q4_0 GGUFs—compressed 4-bit versions of LFM2.5 models (230M, 350M, 1.2B-Instruct, and 2.6B parameters)—trained using Quantization-Aware Distillation, a technique that distills a high-precision teacher model into a quantized student model.
Why it matters
The models retain 97% of their full-precision (BF16) accuracy while maintaining the low memory footprint and fast inference speed of standard 4-bit compression. Developers can now run LFM2.5 on edge hardware—laptops, phones, embedded devices—without the typical quality loss that comes with aggressive compression.
What to watch
The checkpoints are available now on Hugging Face for use with llama.cpp and other GGUF-compatible runtimes. Real-world decode throughput on MacBook Pro, NucBox EVO-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5 shows the 230M and 350M models deliver 4–33% faster throughput than Q5_K_M compressed models at comparable quality.
Ask the AI about this article →
Quantization—compressing neural networks to use fewer bits per weight—is a standard way to shrink models for edge deployment, but it typically degrades accuracy. Liquid AI's approach uses Quantization-Aware Distillation to sidestep this tradeoff: a full-precision teacher model guides the training of a 4-bit student, allowing it to learn patterns that preserve quality despite aggressive compression. The result is that LFM2.5 models in Q4_0 format (a standard 4-bit encoding) recover 96–97% of full-precision accuracy, bridging the gap between efficiency and capability.
The release matters because edge inference—running AI on phones, embedded systems, and small devices rather than in data centers—is constrained by memory and power. By maintaining accuracy while hitting the speed and memory targets of native Q4_0 quantization, these checkpoints make LFM2.5 practical for real-world deployment on heterogeneous hardware: GPU-accelerated devices like MacBooks and NucBox deliver 4–33% faster decoding, while ARM CPU devices like Raspberry Pi and Samsung phones can run the same models at competitive quality.
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytime
Ask AI anything about this article. Q&As are published on this page for other readers too.
