Claude Fable 5.1 builds matrix-free Transformer site, then a 10M Japanese SLM

The site and the SLMs emerged from a single goal-setting workflow. The author, whose graduate work was in parsing and programming-language processing, first asked for a tutorial, then asked whether derivatives could be dropped from the explanation, then asked for a 1M-parameter SLM, then for speeding up, then for scale — each step building on the previous output.

A notable intermediate finding shaped the rest: on the same roughly 7,000-parameter model and corpus, perturbation-based (zeroth-order) training reached loss 1.0 in 62 minutes versus 4 minutes for backprop with Adam — about a 15× gap. That result drove the site's switch to a "no derivatives" framing, with backprop relegated to an appendix. Speed work came before any matrix library: loop reordering lifted 16-thread throughput from 1,430 to 5,350 tokens/sec, then SIMD and Float32 conversion pushed it to 27,700. The author also cautions that Zen 4's AVX-512 splits 512-bit registers across 256-bit units, so the SIMD gain came from fewer instructions, not wider arithmetic.

What the exercise ultimately tests is whether the pedagogy holds up as engineering — whether explaining Transformers without matrices, and training small models without matrix libraries, can still produce recognizable language output. The evidence is mixed but concrete: the 1M model's sentences have structure without meaning, while the 10M model stabilizes speaker-verb agreement and reuses character names within a scene. Whether that arc continues to larger sizes on CPU, or requires the very matrix infrastructure the project set out to avoid, is the open question the author leaves to readers.