Cover Image

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Parmar, Jupinder, Satheesh, Sanjev, Patwary, Mostofa, Shoeybi, Mohammad, Catanzaro, Bryan
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Data, Data Everywhere: A Guide for Pretraining Dataset Construction
by: Parmar, Jupinder, et al.
Published: (2024)

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
by: Mahabadi, Rabeeh Karimi, et al.
Published: (2025)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
by: Akter, Syeda Nahida, et al.
Published: (2024)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
by: Feng, Steven, et al.
Published: (2024)

RLP: Reinforcement as a Pretraining Objective
by: Hatamizadeh, Ali, et al.
Published: (2025)

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
by: Su, Dan, et al.
Published: (2024)

Compact Language Models via Pruning and Knowledge Distillation
by: Muralidharan, Saurav, et al.
Published: (2024)

Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
by: Akter, Syeda Nahida, et al.
Published: (2025)

Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning
by: Lu, Ximing, et al.
Published: (2025)

Don't Throw Away Your Pretrained Model
by: Feng, Shangbin, et al.
Published: (2025)

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
by: Liu, Zihan, et al.
Published: (2024)

InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
by: Wang, Boxin, et al.
Published: (2023)

On Data Engineering for Scaling LLM Terminal Capabilities
by: Pi, Renjie, et al.
Published: (2026)

RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
by: Huang, Jie, et al.
Published: (2023)

Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
by: Taghibakhshi, Ali, et al.
Published: (2025)

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning
by: Jung, Jaehun, et al.
Published: (2025)

Don't Pay Attention
by: Hammoud, Mohammad, et al.
Published: (2025)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
by: Xu, Chejian, et al.
Published: (2025)

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
by: Taghibakhshi, Ali, et al.
Published: (2025)

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
by: Lee, Chankyu, et al.
Published: (2024)

Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
by: Parmar, Mihir, et al.
Published: (2022)

Think, But Don't Overthink: Reproducing Recursive Language Models
by: Wang, Daren
Published: (2026)

Upcycling Large Language Models into Mixture of Experts
by: He, Ethan, et al.
Published: (2024)

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
by: Roy, Debjyoti Saha, et al.
Published: (2024)

Language Models Don't Learn the Physical Manifestation of Language
by: Lee, Bruce W., et al.
Published: (2024)

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
by: Feng, Tao, et al.
Published: (2025)

AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
by: Liu, Zihan, et al.
Published: (2025)

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
by: Lin, Sheng-Chieh, et al.
Published: (2024)

What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization
by: Rezaeimanesh, Sara, et al.
Published: (2026)

Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models
by: Park, Seungcheol, et al.
Published: (2023)

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
by: Xu, Peng, et al.
Published: (2024)

ChatQA: Surpassing GPT-4 on Conversational QA and RAG
by: Liu, Zihan, et al.
Published: (2024)

Nemotron-4 15B Technical Report
by: Parmar, Jupinder, et al.
Published: (2024)

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
by: Hu, Zhiyuan, et al.
Published: (2024)

Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models
by: Lee, Jonggeun, et al.
Published: (2025)

Retrieval meets Long Context Large Language Models
by: Xu, Peng, et al.
Published: (2023)

Music Flamingo: Scaling Music Understanding in Audio Language Models
by: Ghosh, Sreyan, et al.
Published: (2025)

AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
by: Chen, Yang, et al.
Published: (2025)

Don't Touch My Diacritics
by: Gorman, Kyle, et al.
Published: (2024)

Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Large Language Models
by: Yao, Lin
Published: (2026)