Cover Image

Replaying pre-training data improves fine-tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kotha, Suhas, Liang, Percy
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Pre-training under infinite compute
by: Kim, Konwoo, et al.
Published: (2025)

Data-efficient pre-training by scaling synthetic megadocs
by: Kim, Konwoo, et al.
Published: (2026)

Understanding Catastrophic Forgetting in Language Models via Implicit Inference
by: Kotha, Suhas, et al.
Published: (2023)

Testing the Limits of Jailbreaking Defenses with the Purple Problem
by: Kim, Taeyoun, et al.
Published: (2024)

Repetition Improves Language Model Embeddings
by: Springer, Jacob Mitchell, et al.
Published: (2024)

LayerNorm: A key component in parameter-efficient fine-tuning
by: ValizadehAslani, Taha, et al.
Published: (2024)

Complexity-aware fine-tuning
by: Goncharov, Andrey, et al.
Published: (2025)

Prioritized Replay for RL Post-training
by: Fatemi, Mehdi
Published: (2026)

Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs
by: Zhong, Ziqian, et al.
Published: (2025)

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
by: Lee, Tony, et al.
Published: (2026)

You can remove GPT2's LayerNorm by fine-tuning
by: Heimersheim, Stefan
Published: (2024)

The representation landscape of few-shot learning and fine-tuning in large language models
by: Doimo, Diego, et al.
Published: (2024)

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
by: Liu, Hong, et al.
Published: (2023)

The more polypersonal the better -- a short look on space geometry of fine-tuned layers
by: Kudriashov, Sergei, et al.
Published: (2025)

Methods of improving LLM training stability
by: Rybakov, Oleg, et al.
Published: (2024)

Deep literature reviews: an application of fine-tuned language models to migration research
by: Iacus, Stefano M., et al.
Published: (2025)

Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?
by: Sun, Albert Yu, et al.
Published: (2023)

Rethinking harmless refusals when fine-tuning foundation models
by: Pop, Florin, et al.
Published: (2024)

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
by: Sun, Yifan, et al.
Published: (2025)

Multi-task retriever fine-tuning for domain-specific and efficient RAG
by: Béchard, Patrice, et al.
Published: (2025)

Uncertainty quantification in fine-tuned LLMs using LoRA ensembles
by: Balabanov, Oleksandr, et al.
Published: (2024)

Sample-efficient LLM Optimization with Reset Replay
by: Liu, Zichuan, et al.
Published: (2025)

Independence Tests for Language Models
by: Zhu, Sally, et al.
Published: (2025)

Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores
by: Lu, Jun, et al.
Published: (2024)

Improving RL Exploration for LLM Reasoning through Retrospective Replay
by: Dou, Shihan, et al.
Published: (2025)

Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
by: Spiegelhalter, Urs, et al.
Published: (2025)

On the Entropy Calibration of Language Models
by: Cao, Steven, et al.
Published: (2025)

Machine-assisted writing evaluation: Exploring pre-trained language models in analyzing argumentative moves
by: Qin, Wenjuan, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
by: Ma, Weiyu, et al.
Published: (2026)

What explains the success of cross-modal fine-tuning with ORCA?
by: García-de-Herreros, Paloma, et al.
Published: (2024)

Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
by: Jang, Jongseong, et al.
Published: (2022)

Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems
by: Lakatos, Robert, et al.
Published: (2024)

Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG
by: Moreira, Gabriel de Souza P., et al.
Published: (2024)

Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation
by: Burns, Thomas F, et al.
Published: (2025)

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
by: Zhou, Huichi, et al.
Published: (2025)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)

RePO: Replay-Enhanced Policy Optimization
by: Li, Siheng, et al.
Published: (2025)

Adaptive Memory Replay for Continual Learning
by: Smith, James Seale, et al.
Published: (2024)

Blackbox Model Provenance via Palimpsestic Membership Inference
by: Kuditipudi, Rohith, et al.
Published: (2025)