DualOpt: A decoupled optimization approach tailored separately for training neural networks from scratch and fine-tuning pre-trained models

Researchers propose DualOpt, a novel optimizer that uses different strategies for two training paradigms: real-time layer-wise weight decay for training from scratch, and weight rollback integrated into the optimizer for fine-tuning pre-trained models.

For fine-tuning, weight rollback is incorporated into each weight update step to maintain consistency in weight distribution between upstream and downstream models, mitigating knowledge forgetting. Layer-wise weight decay dynamically adjusts rollback levels across layers to adapt to varying downstream task demands.

Experiments across image classification, object detection, semantic segmentation, and instance segmentation demonstrate state-of-the-art performance. Code is available at the provided URL.