AIToday
Large Language ModelsarXiv cs.CVPublished: Apr 28, 2026, 13:00 JST

DualOpt: A decoupled optimization approach tailored separately for training neural networks from scratch and fine-tuning pre-trained models

Send on LINE

3 Key Points

  1. Researchers propose DualOpt, a novel optimizer that uses different strategies for two training paradigms: real-time layer-wise weight decay for training from scratch, and weight rollback integrated into the optimizer for fine-tuning pre-trained models.

  2. For fine-tuning, weight rollback is incorporated into each weight update step to maintain consistency in weight distribution between upstream and downstream models, mitigating knowledge forgetting. Layer-wise weight decay dynamically adjusts rollback levels across layers to adapt to varying downstream task demands.

  3. Experiments across image classification, object detection, semantic segmentation, and instance segmentation demonstrate state-of-the-art performance. Code is available at the provided URL.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleJury selection in Musk v. Altman case begins; prospective jurors express negative views of Elon Musk