QLoRA's default learning rate may trap small-dataset fine-tuning

A practitioner shared their experience with QLoRA fine-tuning on small datasets, revealing a persistent mismatch between published guidance and real-world outcomes. The standard learning rate of 2e-4, recommended across tutorials and documentation including the Unsloth docs and Hugging Face examples, originates from the Alpaca dataset's 52k samples — a scale far larger than the 5–10k sample datasets most practitioners assemble themselves.

When applied to smaller datasets, the 2e-4 rate triggers overfitting within the first epoch, a problem that can be difficult to diagnose. The author describes the symptom: training loss decreases smoothly, creating the appearance of progress, while evaluation loss remains static or climbs, indicating the model is memorizing the training set rather than learning generalizable patterns. This pattern repeated across multiple training runs, generating what the author calls "seven identical bad evals in a row."

The debugging process consumed significant effort over three weeks. The author recleaned the dataset twice, rewrote the prompt template twice, and spent an entire day hand-relabeling rows (reducing a starting set of 8k rows to approximately 7200 after removing low-quality examples). None of these interventions improved evaluation performance because the underlying issue was the learning rate itself.

The breakthrough came when the author changed two hyperparameters: reducing the learning rate to 1e-4 and increasing epochs from 3 to 5. This adjustment produced an immediate and substantial improvement in evaluation loss — described as jumping "more than everything else combined." The post implicitly argues that practitioners fine-tuning on small, custom datasets should not assume the published defaults are optimal and may need to experiment with lower learning rates to avoid overfitting.

The QLoRA learning rate of 2e-4 has become the de facto standard across tutorials and library documentation, anchored to the Alpaca fine-tuning experiments that used 52k samples. However, most practitioners today work with much smaller custom datasets — typically 5–10k samples — scraped and labeled in-house. At this scale, the 2e-4 rate appears to trigger rapid overfitting within a single epoch, causing training loss to fall smoothly while evaluation loss remains flat or rises, a pattern that can persist across multiple training runs and mask the root cause.

The author's experience illustrates the cost of this mismatch: three weeks of iterative debugging (reprocessing data twice, rewriting prompts twice, hand-relabeling rows) yielded no improvement because the fundamental hyperparameter was misaligned with the dataset size. Only when the learning rate was halved and epochs were extended did the model begin to generalize properly. This suggests that the current industry guidance, while valid for Alpaca-scale datasets, may systematically mislead practitioners working at smaller scales.