Tiny LLM hits 573,952 params, repeat-mumbles at 18000字

The project was framed from the start as a record of an experiment rather than a tutorial, and the author set three hard constraints: no external code libraries, a single CPU thread, and everything self-contained. The stated reason for the CPU limit was that parallel processing would blur the study of how an LLM works, while the from-scratch rule reflected both curiosity and the fact that no existing library targets such a stripped-down setup.

The model itself is deliberately simple, with three layers, sigmoid activation, and argmax token selection, trained by backpropagation and gradient descent with mean squared error. The early results show memorization working on short inputs, but as the training text grew the output degraded into repeated characters, which the author connects to the 16-token context and a scoring method that does not discourage repetition.

The stakes here are personal rather than commercial. The author describes the work as about two days driven mostly by curiosity, and the failures he lists, such as weight matrices initialized to 1.0 and a context too short to see enough Japanese characters, are the kind of practical lessons that may carry into the next model he says he intends to build.