28.9M-parameter LLM runs on $8 ESP32-S3 microcontroller

A developer created a 28.9 million parameter language model running on an ESP32-S3 microcontroller, a chip costing about $8. The model generates text directly on the device at roughly 9 tokens per second, with nothing sent to a server, and displays each word on a small wired screen. This represents roughly a hundred-fold increase in model size compared to the previous record on the same hardware class, which held only 260 thousand parameters.

The breakthrough relies on Google's Per-Layer Embeddings technique, originally designed for Gemma models. In standard language models, all parameters must be accessible from fast memory, but the ESP32-S3 has only 512KB of SRAM—a severe constraint. Per-Layer Embeddings solves this by recognizing that most language model parameters sit in embedding lookup tables that the model reads rather than computes. The developer keeps the 25 million row embedding table in the chip's 16MB flash memory (slow but large) and loads only the rows needed for each token, roughly 450 bytes at a time. The small part that does actual reasoning stays in fast memory, making the large model nearly cost-free to run because most of it is never loaded. The final model is 14.9MB at 4-bit quantization.

The model was trained on TinyStories, a dataset of short synthetic stories simple enough for small models to learn to write coherently. As a result, it generates short stories and maintains coherence but cannot answer questions, follow instructions, write code, or provide factual information. The developer emphasizes that the significance lies in the architecture and memory layout innovation rather than the model's capabilities. The firmware, wiring instructions, training code, ablations, and quantization code are available in the project repository, including the full method writeup in RESULTS.md. The developer deliberately preserved the commit history, including a bug in early parameter accounting and its correction, to show how the numbers moved and why. The work builds on Andrej Karpathy's llama2.c project, which established that small language models can be trained and run in plain C.

The achievement represents a shift in how language models can be deployed at the hardware extreme. Traditionally, fitting models onto microcontrollers required severe parameter reduction because the entire model had to fit into fast memory—the ESP32-S3's 512KB SRAM—which is why prior models on this chip class maxed out at 260 thousand parameters. By adopting Google's Per-Layer Embeddings architecture, the developer decoupled the model's lookup tables (embeddings) from its reasoning core, allowing 25 million of the 28.9 million parameters to live in the chip's slower 16MB flash memory. Only the active computation—the "thinking" part—needs to reside in fast memory, fundamentally changing the memory constraint equation.

The model achieves this while maintaining practical performance: 9 tokens per second is sufficient for real-time text generation on a screen, and coherent short-story generation on TinyStories proves the architectural trade-off does not destroy reasoning quality at scale. The deliberate choice to train on TinyStories acknowledges the model's limitations (no question-answering, instruction-following, or factual knowledge), anchoring expectations to what the architecture can actually deliver. This is explicitly about demonstrating that large models can run on tiny edge hardware, not about competing with general-purpose AI systems.

No comments yet. Be the first to share your thoughts!