LoRA tuning of Qwen2.5-1.5B on small Japanese data: format beats volume

The piece is written from direct experience rather than as a product announcement. In the configuration the author actually tried, the base model's weights stayed frozen and only a small set of additional parameters was trained, which the author describes as a relatively manageable procedure.

The guide is explicit about where this approach fits. Rather than loading the model with large amounts of new knowledge, it is suited to aligning responses to a specific format, matching in-house terminology, handling a limited set of FAQ-style questions, and unifying tone and honorific style. With only a few examples, the author warns, attempting to teach a broad domain can damage the original model's capabilities.

Practical caution runs through the steps: mixed polite and plain styles in the training data tend to produce wavering output, duplicate or contradictory answers hit small datasets especially hard, and answers should be checked after training because a high learning rate encourages rote memorization. Evaluation is split into identical questions, rephrased similar questions, and general questions absent from the training data — the last a check that the model has not degraded. The suggested starting point is a few dozen clean samples, with data volume and model size increased only after the output has been reviewed.