Cover Image

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Nguyen, Thao, Li, Yang, Golovneva, Olga, Zettlemoyer, Luke, Oh, Sewoong, Schmidt, Ludwig, Li, Xian
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Better Alignment with Instruction Back-and-Forth Translation
by: Nguyen, Thao, et al.
Published: (2024)

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
by: He, Jacqueline, et al.
Published: (2026)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
by: Chen, Tong, et al.
Published: (2025)

Self-Improving Pretraining: using post-trained models to pretrain better models
by: Tan, Ellen Xiaoqing, et al.
Published: (2026)

Privacy-Preserving Instructions for Aligning Large Language Models
by: Yu, Da, et al.
Published: (2024)

Sampling from Your Language Model One Byte at a Time
by: Hayase, Jonathan, et al.
Published: (2025)

Evaluating Copyright Takedown Methods for Language Models
by: Wei, Boyi, et al.
Published: (2024)

Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation
by: Iyer, Vivek, et al.
Published: (2024)

Is Training Data Quality or Quantity More Impactful to Small Language Model Performance?
by: Sajith, Aryan, et al.
Published: (2024)

Multi-Token Attention
by: Golovneva, Olga, et al.
Published: (2025)

CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models
by: Irwin, Lucas, et al.
Published: (2025)

A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification
by: Calbucura, Nicolas, et al.
Published: (2025)

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
by: He, Nan, et al.
Published: (2024)

Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer
by: Lee, Seungyoon, et al.
Published: (2025)

DataMan: Data Manager for Pre-training Large Language Models
by: Peng, Ru, et al.
Published: (2025)

QoSBERT: An Uncertainty-Aware Approach based on Pre-trained Language Models for Service Quality Prediction
by: Wang, Ziliang, et al.
Published: (2025)

Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models
by: Zhuang, Xinlin, et al.
Published: (2025)

Contextual Position Encoding: Learning to Count What's Important
by: Golovneva, Olga, et al.
Published: (2024)

(Mis)Fitting: A Survey of Scaling Laws
by: Li, Margaret, et al.
Published: (2025)

Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training
by: Wu, Linjuan, et al.
Published: (2025)

Demystifying Prompts in Language Models via Perplexity Estimation
by: Gonen, Hila, et al.
Published: (2022)

Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders
by: Xin, Yuan, et al.
Published: (2024)

Projective Methods for Mitigating Gender Bias in Pre-trained Language Models
by: Dawkins, Hillary, et al.
Published: (2024)

A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models
by: Hayat, Ahatsham, et al.
Published: (2024)

Probing Language Models for Pre-training Data Detection
by: Liu, Zhenhua, et al.
Published: (2024)

From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
by: Li, Ming, et al.
Published: (2023)

SongSage: A Large Musical Language Model with Lyric Generative Pre-training
by: Guo, Jiani, et al.
Published: (2026)

Micro Language Models Enable Instant Responses
by: Cheng, Wen, et al.
Published: (2026)

Relational Prompt-based Pre-trained Language Models for Social Event Detection
by: Li, Pu, et al.
Published: (2024)

Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models
by: Liang, Haoyu, et al.
Published: (2026)

SuperBPE: Space Travel for Language Models
by: Liu, Alisa, et al.
Published: (2025)

Multilingual Diversity Improves Vision-Language Representations
by: Nguyen, Thao, et al.
Published: (2024)

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
by: Blevins, Terra, et al.
Published: (2024)

Examining Forgetting in Continual Pre-training of Aligned Large Language Models
by: Li, Chen-An, et al.
Published: (2024)

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism
by: Li, Guanchen, et al.
Published: (2024)

Reverse Training to Nurse the Reversal Curse
by: Golovneva, Olga, et al.
Published: (2024)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
by: Hayase, Jonathan, et al.
Published: (2024)

Reusing Pre-Training Data at Test Time is a Compute Multiplier
by: Fang, Alex, et al.
Published: (2025)

KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
by: Li, Yudong, et al.
Published: (2026)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)