Cover Image

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Marek, Martin, Cho, Dongkyu, Qiu, Shikai, Chunara, Rumi, Izmailov, Pavel, Wilson, Andrew Gordon
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dealing with the Evil Twins: Improving Random Augmentation by Addressing Catastrophic Forgetting of Diverse Augmentations
by: Cho, Dongkyu, et al.
Published: (2025)

Forget Forgetting: Continual Learning in a World of Abundant Memory
by: Cho, Dongkyu, et al.
Published: (2025)

Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration
by: Cho, Dongkyu, et al.
Published: (2025)

Tree of Concepts: Interpretable Continual Learners in Non-Stationary Clinical Domains
by: Cho, Dongkyu, et al.
Published: (2026)

From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
by: Finzi, Marc, et al.
Published: (2026)

Do Clinical Models Change Treatment Decisions?
by: Cho, Dongkyu, et al.
Published: (2026)

Disparate Effect Of Missing Mediators On Transportability of Causal Effects
by: Mhasawade, Vishwali, et al.
Published: (2024)

Can a Confident Prior Replace a Cold Posterior?
by: Marek, Martin, et al.
Published: (2024)

Large Language Models Are Zero-Shot Time Series Forecasters
by: Gruver, Nate, et al.
Published: (2023)

Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
by: Qiu, Shikai, et al.
Published: (2025)

Generalization in Healthcare AI: Evaluation of a Clinical Large Language Model
by: Rahman, Salman, et al.
Published: (2024)

Compute Better Spent: Replacing Dense Layers with Structured Matrices
by: Qiu, Shikai, et al.
Published: (2024)

Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
by: Qiu, Shikai, et al.
Published: (2025)

Configurable Fairness: Direct Optimization of Parity Metrics via Vision-Language Models
by: Zhang, Miao, et al.
Published: (2024)

Understanding Disparities in Post Hoc Machine Learning Explanation
by: Mhasawade, Vishwali, et al.
Published: (2024)

Transferring Knowledge from Large Foundation Models to Small Downstream Models
by: Qiu, Shikai, et al.
Published: (2024)

Domain Adaptation Under MNAR Missingness
by: Stokes, Tyrel, et al.
Published: (2025)

Impact on Public Health Decision Making by Utilizing Big Data Without Domain Knowledge
by: Zhang, Miao, et al.
Published: (2024)

Replay Can Provably Increase Forgetting
by: Mahdaviyeh, Yasaman, et al.
Published: (2025)

Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgetting
by: Qiu, Suming, et al.
Published: (2025)

Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
by: Marek, Martin, et al.
Published: (2025)

Customizing the Inductive Biases of Softmax Attention using Structured Matrices
by: Kuang, Yilun, et al.
Published: (2025)

When Forgetting Builds Reliability: LLM Unlearning for Reliable Hardware Code Generation
by: Liang, Yiwen, et al.
Published: (2025)

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
by: Wiemann, Matt L., et al.
Published: (2026)

FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
by: Feng, Yujie, et al.
Published: (2026)

Spurious Forgetting in Continual Learning of Language Models
by: Zheng, Junhao, et al.
Published: (2025)

In-Context Clustering with Large Language Models
by: Wang, Ying, et al.
Published: (2025)

Match or Replay: Self Imitating Proximal Policy Optimization
by: Chaudhary, Gaurav, et al.
Published: (2026)

Implicit Updates for Average-Reward Temporal Difference Learning
by: Kim, Hwanwoo, et al.
Published: (2025)

Deep Learning is Not So Mysterious or Different
by: Wilson, Andrew Gordon
Published: (2025)

CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay
by: Zhang, Jianshu, et al.
Published: (2024)

Out-of-Distribution Detection Methods Answer the Wrong Questions
by: Li, Yucen Lily, et al.
Published: (2025)

Continual Learning: Less Forgetting, More OOD Generalization via Adaptive Contrastive Replay
by: Rezaei, Hossein, et al.
Published: (2024)

SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay
by: Yoon, Hyungjun, et al.
Published: (2024)

Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
by: Lotfi, Sanae, et al.
Published: (2024)

Language Generation with Replay: A Learning-Theoretic View of Model Collapse
by: Racca, Giorgio, et al.
Published: (2026)

Non-Vacuous Generalization Bounds for Large Language Models
by: Lotfi, Sanae, et al.
Published: (2023)

Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices
by: Potapczynski, Andres, et al.
Published: (2024)

Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay
by: Zhou, Yuhang, et al.
Published: (2024)

Replay-and-Forget-Free Graph Class-Incremental Learning: A Task Profiling and Prompting Approach
by: Niu, Chaoxi, et al.
Published: (2024)