Cover Image

MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Yupeng, Wang, Senmiao, Zhang, Yushun, Lin, Zhihang, Zhang, Haozhe, Sun, Weijian, Ding, Tian, Sun, Ruoyu
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective
by: Wang, Senmiao, et al.
Published: (2025)

VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
by: Gong, Xuan, et al.
Published: (2025)

Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting
by: Ding, Fei, et al.
Published: (2025)

Why Transformers Need Adam: A Hessian Perspective
by: Zhang, Yushun, et al.
Published: (2024)

Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
by: Zhang, Zhixin, et al.
Published: (2025)

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
by: Fawi, Muhammad
Published: (2024)

Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
by: Lai, Song, et al.
Published: (2025)

Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
by: Sanyal, Sunny, et al.
Published: (2025)

Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
by: Diao, Muxi, et al.
Published: (2026)

Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
by: Schweighofer, Kajetan, et al.
Published: (2026)

Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
by: Wu, Chao-Chung, et al.
Published: (2025)

Preserving Diversity in Supervised Fine-Tuning of Large Language Models
by: Li, Ziniu, et al.
Published: (2024)

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
by: Fernando, Heshan, et al.
Published: (2024)

ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
by: Wang, Ruoyu, et al.
Published: (2024)

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
by: Ran, Yide, et al.
Published: (2025)

Preventing Catastrophic Forgetting: Behavior-Aware Sampling for Safer Language Model Fine-Tuning
by: Pham, Anh, et al.
Published: (2025)

Bridging the Gap: Self-Optimized Fine-Tuning for LLM-based Recommender Systems
by: Tang, Heng, et al.
Published: (2025)

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
by: Yan, Peishen, et al.
Published: (2026)

DeMo: Decoupled Momentum Optimization
by: Peng, Bowen, et al.
Published: (2024)

PDHG-Unrolled Learning-to-Optimize Method for Large-Scale Linear Programming
by: Li, Bingheng, et al.
Published: (2024)

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
by: Li, Ziniu, et al.
Published: (2023)

Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
by: Abdi, Hossein, et al.
Published: (2025)

Adam-mini: Use Fewer Learning Rates To Gain More
by: Zhang, Yushun, et al.
Published: (2024)

RL Fine-Tuning Heals OOD Forgetting in SFT
by: Jin, Hangzhan, et al.
Published: (2025)

DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning
by: Zhang, Junbo, et al.
Published: (2026)

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
by: Ju, Yiming, et al.
Published: (2024)

Secure LLM Fine-Tuning via Safety-Aware Probing
by: Wu, Chengcan, et al.
Published: (2025)

Forget the Data and Fine-Tuning! Just Fold the Network to Compress
by: Wang, Dong, et al.
Published: (2025)

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
by: Wang, Fangxin, et al.
Published: (2026)

RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
by: Liu, Jun, et al.
Published: (2025)

Catastrophic Forgetting Mitigation via Discrepancy-Weighted Experience Replay
by: Xu, Xinrun, et al.
Published: (2025)

FedRef: Bayesian Fine-Tuning using a Reference Model to Mitigate Catastrophic Forgetting for Heterogeneous Federated Learning
by: Yoon, Taehwan, et al.
Published: (2025)

Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
by: Li, Ziniu, et al.
Published: (2025)

CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications
by: Cao, Yupeng, et al.
Published: (2024)

Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation
by: Liu, Jinmei, et al.
Published: (2026)

Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection
by: Lai, Zhixin, et al.
Published: (2024)

Exact Causal Attention with 10% Fewer Operations
by: Rybin, Dmitry, et al.
Published: (2025)

Revisiting Catastrophic Forgetting in Large Language Model Tuning
by: Li, Hongyu, et al.
Published: (2024)

In-Context Learning State Vector with Inner and Momentum Optimization
by: Li, Dongfang, et al.
Published: (2024)

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
by: Wang, Shumin, et al.
Published: (2026)