Researchers reveal that fine-tuning method and learning rate together determine whether AI vision models forget old knowledge — LoRA stays more balanced than full retraining

Researchers at arXiv ran 80 controlled experiments comparing two ways to adapt CLIP (an AI model that matches images to text descriptions) to new tasks — Full Fine-Tuning (retraining all weights) versus LoRA (a lightweight method that only adjusts specific parameters). They tested the same learning rates across both methods on satellite imagery and pet photos to isolate which technique causes the model to "forget" its original capabilities.

Full Fine-Tuning dramatically changed how the model's attention patterns worked depending on learning rate: at low rates it made small adjustments, but at higher rates it drastically restructured attention, suggesting the model was overwriting foundational knowledge. LoRA, by contrast, kept attention patterns stable across all learning rates, implying it learns new tasks without erasing what the model already knew.

For data scientists and engineers choosing between fine-tuning methods: if your AI model needs to stay useful on multiple tasks (answering questions about satellite maps AND recognizing household pets), LoRA appears safer because it doesn't force you to rebuild internal knowledge structures. Full Fine-Tuning wins on single-task accuracy but at the cost of breaking performance on unrelated problems — a hidden tradeoff that matched learning-rate experiments now make visible.