Image de couverture de livre

Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Fang, Alex, Pouransari, Hadi, Jordan, Matt, Toshev, Alexander, Shankar, Vaishaal, Schmidt, Ludwig, Gunter, Tom
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

MobileCLIP2: Improving Multi-Modal Reinforced Training
par: Faghri, Fartash, et autres
Publié: (2025)

Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum
par: Pouransari, Hadi, et autres
Publié: (2024)

Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
par: Zhang, Yuhui, et autres
Publié: (2023)

TiC-CLIP: Continual Training of CLIP Models
par: Garg, Saurabh, et autres
Publié: (2023)

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
par: Li, Jeffrey, et autres
Publié: (2025)

OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
par: Ngo, Huong, et autres
Publié: (2025)

Pretraining with hierarchical memories: separating long-tail and common knowledge
par: Pouransari, Hadi, et autres
Publié: (2025)

DataComp-LM: In search of the next generation of training sets for language models
par: Li, Jeffrey, et autres
Publié: (2024)

Reusing Pre-Training Data at Test Time is a Compute Multiplier
par: Fang, Alex, et autres
Publié: (2025)

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2023)

Language Models Improve When Pretraining Data Matches Target Tasks
par: Mizrahi, David, et autres
Publié: (2025)

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
par: Nguyen, Thao, et autres
Publié: (2025)

Large Scale Transfer Learning for Tabular Data via Language Modeling
par: Gardner, Josh, et autres
Publié: (2024)

Repetition Improves Language Model Embeddings
par: Springer, Jacob Mitchell, et autres
Publié: (2024)

Language models scale reliably with over-training and on downstream tasks
par: Gadre, Samir Yitzhak, et autres
Publié: (2024)

CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2024)

FairFlow: Mitigating Dataset Biases through Undecided Learning
par: Cheng, Jiali, et autres
Publié: (2025)

Repetitions are not all alike: distinct mechanisms sustain repetition in language models
par: Mahaut, Matéo, et autres
Publié: (2025)

QuRating: Selecting High-Quality Data for Training Language Models
par: Wettig, Alexander, et autres
Publié: (2024)

MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries
par: Elgaar, Mohamed, et autres
Publié: (2024)

Watermarking Needs Input Repetition Masking
par: Khachaturov, David, et autres
Publié: (2025)

FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
par: Hsieh, Cheng-Yu, et autres
Publié: (2025)

Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models
par: Paech, Samuel, et autres
Publié: (2025)

Resolving Discrepancies in Compute-Optimal Scaling of Language Models
par: Porian, Tomer, et autres
Publié: (2024)

GATES: Self-Distillation under Privileged Context with Consensus Gating
par: Stein, Alex, et autres
Publié: (2026)

Prompt Repetition Improves Non-Reasoning LLMs
par: Leviathan, Yaniv, et autres
Publié: (2025)

Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing
par: Wang, Weichuan, et autres
Publié: (2024)

Learning to Reason for Hallucination Span Detection
par: Su, Hsuan, et autres
Publié: (2025)

FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
par: Kirchenbauer, John, et autres
Publié: (2025)

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition
par: Duan, Yifei, et autres
Publié: (2025)

Large Language Models as Generalizable Policies for Embodied Tasks
par: Szot, Andrew, et autres
Publié: (2023)

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
par: Saada, Thiziri Nait, et autres
Publié: (2025)

Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
par: Li, Jeffrey, et autres
Publié: (2026)

EqualizeIR: Mitigating Linguistic Biases in Retrieval Models
par: Cheng, Jiali, et autres
Publié: (2025)

HuCurl: Human-induced Curriculum Discovery
par: Elgaar, Mohamed, et autres
Publié: (2023)

BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks
par: Rodriguez, Juan, et autres
Publié: (2024)

Olmix: A Framework for Data Mixing Throughout LM Development
par: Chen, Mayee F., et autres
Publié: (2026)

Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization
par: Lu, Yen-Ju, et autres
Publié: (2025)

LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
par: Askari, Hadi, et autres
Publié: (2025)

Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents
par: Miao, Ziyang, et autres
Publié: (2025)