Image de couverture de livre

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Bethune, Louis, Grangier, David, Busbridge, Dan, Gualdoni, Eleonora, Cuturi, Marco, Ablin, Pierre
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
par: Saada, Thiziri Nait, et autres
Publié: (2025)

Scaling Laws for Optimal Data Mixtures
par: Shukor, Mustafa, et autres
Publié: (2025)

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
par: Gualdoni, Eleonora, et autres
Publié: (2026)

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
par: Ablin, Pierre, et autres
Publié: (2025)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
par: Grangier, David, et autres
Publié: (2024)

Dynamic Gradient Alignment for Online Data Mixing
par: Fan, Simin, et autres
Publié: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
par: Sedova, Anastasiia, et autres
Publié: (2026)

Optimal Splitting of Language Models from Mixtures to Specialized Domains
par: Seto, Skyler, et autres
Publié: (2026)

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
par: Mlodozeniec, Bruno, et autres
Publié: (2025)

The Geometries of Truth Are Orthogonal Across Tasks
par: Azizian, Waiss, et autres
Publié: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
par: Monteiro, João, et autres
Publié: (2026)

Need a Small Specialized Language Model? Plan Early!
par: Grangier, David, et autres
Publié: (2024)

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
par: Kirchhof, Michael, et autres
Publié: (2024)

Scaling Categorical Flow Maps
par: Davis, Oscar, et autres
Publié: (2026)

Distillation Scaling Laws
par: Busbridge, Dan, et autres
Publié: (2025)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
par: Sakamoto, Keitaro, et autres
Publié: (2026)

ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
par: Longpre, Shayne, et autres
Publié: (2025)

Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures
par: Vesseron, Nina, et autres
Publié: (2025)

The AdEMAMix Optimizer: Better, Faster, Older
par: Pagliardini, Matteo, et autres
Publié: (2024)

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
par: Gupta, Prakhar, et autres
Publié: (2026)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
par: Krajewski, Jakub, et autres
Publié: (2025)

Pretraining with hierarchical memories: separating long-tail and common knowledge
par: Pouransari, Hadi, et autres
Publié: (2025)

Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
par: Xie, Wanyun, et autres
Publié: (2025)

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
par: Zhang, Biao, et autres
Publié: (2024)

Multivariate Conformal Prediction using Optimal Transport
par: Klein, Michal, et autres
Publié: (2025)

Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training
par: Reynolds, John Graham
Publié: (2025)

Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
par: Watts, Ishaan, et autres
Publié: (2026)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
par: Feng, Jiahai, et autres
Publié: (2024)

From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
par: Zhao, Eric, et autres
Publié: (2025)

Improving Language Plasticity via Pretraining with Active Forgetting
par: Chen, Yihong, et autres
Publié: (2023)

Learning Unmasking Policies for Diffusion Language Models
par: Jazbec, Metod, et autres
Publié: (2025)

HyperTransport: Amortized Conditioning of T2I Generative Models
par: Maiorca, Valentino, et autres
Publié: (2026)

Training Bilingual LMs with Data Constraints in the Targeted Language
par: Seto, Skyler, et autres
Publié: (2024)

Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe
par: You, Chong, et autres
Publié: (2025)

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
par: Singh, Karan, et autres
Publié: (2026)

Predicting Emergent Capabilities by Finetuning
par: Snell, Charlie, et autres
Publié: (2024)

Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
par: Filippova, Anastasiia, et autres
Publié: (2026)

No Need to Talk: Asynchronous Mixture of Language Models
par: Filippova, Anastasiia, et autres
Publié: (2024)

TaCo: Targeted Concept Erasure Prevents Non-Linear Classifiers From Detecting Protected Attributes
par: Jourdan, Fanny, et autres
Publié: (2023)

Why do objects have many names? A study on word informativeness in language use and lexical systems
par: Gualdoni, Eleonora, et autres
Publié: (2024)