Cover Image

LoRA Learns Less and Forgets Less

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Biderman, Dan, Portes, Jacob, Ortiz, Jose Javier Gonzalez, Paul, Mansheej, Greengard, Philip, Jennings, Connor, King, Daniel, Havens, Sam, Chiley, Vitaliy, Frankle, Jonathan, Blakeney, Cody, Cunningham, John P.
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Does your data spark joy? Performance gains from domain upsampling at the end of training
by: Blakeney, Cody, et al.
Published: (2024)

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
by: Han, Jiayi, et al.
Published: (2024)

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
by: Portes, Jacob, et al.
Published: (2023)

LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
by: Guo, Han, et al.
Published: (2023)

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
by: Sardana, Nikhil, et al.
Published: (2023)

Sparse Upcycling: Inference Inefficient Finetuning
by: Doubov, Sasha, et al.
Published: (2024)

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
by: Ankner, Zachary, et al.
Published: (2024)

UnGuide: Learning to Forget with LoRA-Guided Diffusion Models
by: Polowczyk, Agnieszka, et al.
Published: (2025)

S-LoRA: Serving Thousands of Concurrent LoRA Adapters
by: Sheng, Ying, et al.
Published: (2023)

Long Context RAG Performance of Large Language Models
by: Leng, Quinn, et al.
Published: (2024)

Soup to go: mitigating forgetting during continual learning with model averaging
by: Kleiman, Anat, et al.
Published: (2025)

Run LoRA Run: Faster and Lighter LoRA Implementations
by: Cherniuk, Daria, et al.
Published: (2023)

Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
by: Xu, Yixing, et al.
Published: (2025)

Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning
by: Liu, Yezi, et al.
Published: (2025)

Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
by: Koubbi, Hugo, et al.
Published: (2024)

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
by: Xu, Runze, et al.
Published: (2026)

LoRA as Oracle
by: Arazzi, Marco, et al.
Published: (2026)

Vision as LoRA
by: Wang, Han, et al.
Published: (2025)

FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
by: Zhang, Haoran, et al.
Published: (2026)

LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
by: Rajabzadeh, Hossein, et al.
Published: (2026)

TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
by: Jeon, Jeimin, et al.
Published: (2026)

LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
by: Lee, Seungeon, et al.
Published: (2025)

Inference-Friendly Models With MixAttention
by: Rajput, Shashank, et al.
Published: (2024)

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
by: Gupta, Prakhar, et al.
Published: (2026)

OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
by: Xiong, Yifeng, et al.
Published: (2025)

ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads
by: Zuo, Jingwei, et al.
Published: (2026)

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement
by: Bian, Jieming, et al.
Published: (2024)

LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation
by: Zhou, Hongyun, et al.
Published: (2024)

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
by: Yen, Jui-Nan, et al.
Published: (2024)

CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
by: Chen, Guanduo, et al.
Published: (2025)

LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization
by: Smith, Ethan, et al.
Published: (2024)

SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
by: Liu, Jianmin, et al.
Published: (2025)

TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control
by: Cho, Minkyoung, et al.
Published: (2025)

NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion
by: Chen, Chuheng, et al.
Published: (2025)

ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA
by: Dahan, Aviad, et al.
Published: (2026)

A Note on LoRA
by: Fomenko, Vlad, et al.
Published: (2024)

Mixture of LoRA Experts
by: Wu, Xun, et al.
Published: (2024)

Dataset Size Recovery from LoRA Weights
by: Salama, Mohammad, et al.
Published: (2024)

Merge before Forget: A Single LoRA Continual Learning via Continual Merging
by: Qiao, Fuli, et al.
Published: (2025)

Personalized LoRA for Human-Centered Text Understanding
by: Zhang, You, et al.
Published: (2024)