Researcher releases 125M parameter language model trained from scratch with custom tokenizer and open-source SFT framework for community experimentation

Trained a 12-layer 125M parameter causal language model from scratch using custom 16k BPE tokenizer on WikiText-103 and TinyStories datasets

Achieved 6.19 validation perplexity on WikiText-103 after ~92k training steps without relying on GPT-2 initialization or borrowed tokenizers

Created conversational variant using LoRA (rank 8) fine-tuning on DailyDialog dataset (~87k examples) with completion-only masked loss

Released both base model (librarian-base-130m) and instruct variant (Librarian-Instruct-130m) on Hugging Face with SFT framework for others to build on

Project focused on enabling community experimentation with small language models rather than competing with larger 1B+ parameter instruct models