Image de couverture de livre

Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Watts, Ishaan, Li, Catherine, Goyal, Sachin, Springer, Jacob Mitchell, Raghunathan, Aditi
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Understanding Catastrophic Forgetting in Language Models via Implicit Inference
par: Kotha, Suhas, et autres
Publié: (2023)

Mitigating Bias in RAG: Controlling the Embedder
par: Kim, Taeyoun, et autres
Publié: (2025)

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning
par: Springer, Jacob Mitchell, et autres
Publié: (2024)

Mode-Conditioning Unlocks Superior Test-Time Scaling
par: Wu, Chen Henry, et autres
Publié: (2025)

Repetition Improves Language Model Embeddings
par: Springer, Jacob Mitchell, et autres
Publié: (2024)

Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance
par: Goyal, Sachin, et autres
Publié: (2024)

Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training
par: Reynolds, John Graham
Publié: (2025)

T-MARS: Improving Visual Representations by Circumventing Text Feature Learning
par: Maini, Pratyush, et autres
Publié: (2023)

Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs
par: Zhong, Ziqian, et autres
Publié: (2025)

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
par: Fawi, Muhammad
Publié: (2024)

Understanding Finetuning for Factual Knowledge Extraction
par: Ghosal, Gaurav, et autres
Publié: (2024)

ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
par: Zhong, Ziqian, et autres
Publié: (2025)

Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
par: Zhang, Zhixin, et autres
Publié: (2025)

Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
par: Ren, Weijieying, et autres
Publié: (2024)

Evolutionary Strategies lead to Catastrophic Forgetting in LLMs
par: Abdi, Immanuel, et autres
Publié: (2026)

Self-Trained Verification for Training- and Test-Time Self-Improvement
par: Wu, Chen Henry, et autres
Publié: (2026)

A Granular Study of Safety Pretraining under Model Abliteration
par: Agnihotri, Shashank, et autres
Publié: (2025)

Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies
par: Koloski, Boshko, et autres
Publié: (2023)

Overtrained Language Models Are Harder to Fine-Tune
par: Springer, Jacob Mitchell, et autres
Publié: (2025)

Testing the Limits of Jailbreaking Defenses with the Purple Problem
par: Kim, Taeyoun, et autres
Publié: (2024)

Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts
par: Shashank
Publié: (2026)

Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers
par: Kenneweg, Philip, et autres
Publié: (2024)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
par: Hsiao, Chi-Yuan, et autres
Publié: (2025)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
par: Li, Junzhuo, et autres
Publié: (2025)

Understanding the Influence of Synthetic Data for Text Embedders
par: Springer, Jacob Mitchell, et autres
Publié: (2025)

Improving Language Plasticity via Pretraining with Active Forgetting
par: Chen, Yihong, et autres
Publié: (2023)

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
par: Chen, Howard, et autres
Publié: (2025)

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
par: Gupta, Prakhar, et autres
Publié: (2026)

Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
par: Liu, Chi, et autres
Publié: (2026)

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting
par: Gong, Xuchen, et autres
Publié: (2025)

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
par: Nagarajan, Vaishnavh, et autres
Publié: (2025)

A Comparative Empirical Study of Catastrophic Forgetting Mitigation in Sequential Task Adaptation for Continual Natural Language Processing Systems
par: Abrahamyan, Aram, et autres
Publié: (2026)

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
par: Feng, Lawrence, et autres
Publié: (2026)

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
par: Wei, Rongzhe, et autres
Publié: (2025)

Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs
par: Mendu, Sai Krishna, et autres
Publié: (2025)

Learning to Translate from Soft to Hard LLM Prompts
par: Kongsomjit, Pitipat, et autres
Publié: (2026)

BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
par: Chang, Yupeng, et autres
Publié: (2024)

TRAM: Bridging Trust Regions and Sharpness Aware Minimization
par: Sherborne, Tom, et autres
Publié: (2023)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
par: Feng, Jiahai, et autres
Publié: (2024)