Cover Image

CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Brandfonbrener, David, Zhang, Hanlin, Kirsch, Andreas, Schwarz, Jonathan Richard, Kakade, Sham
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Loss-to-Loss Prediction: Scaling Laws for All Datasets
by: Brandfonbrener, David, et al.
Published: (2024)

Repeat After Me: Transformers are Better than State Space Models at Copying
by: Jelassi, Samy, et al.
Published: (2024)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
by: Zhang, Hanlin, et al.
Published: (2026)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
by: Jin, Jikai, et al.
Published: (2025)

The Role of Sparsity for Length Generalization in Transformers
by: Golowich, Noah, et al.
Published: (2025)

Peer-Predictive Self-Training for Language Model Reasoning
by: Feng, Shi, et al.
Published: (2026)

Deconstructing What Makes a Good Optimizer for Language Models
by: Zhao, Rosie, et al.
Published: (2024)

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
by: Qi, Zhenting, et al.
Published: (2024)

EvoLM: In Search of Lost Language Model Training Dynamics
by: Qi, Zhenting, et al.
Published: (2025)

How Does Critical Batch Size Scale in Pre-training?
by: Zhang, Hanlin, et al.
Published: (2024)

Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
by: Song, Yuda, et al.
Published: (2024)

GQ-VAE: A gated quantized VAE for learning variable length tokens
by: Datta, Theo, et al.
Published: (2025)

CoLoR-GAN: Continual Few-Shot Learning with Low-Rank Adaptation in Generative Adversarial Networks
by: Ali, Munsif, et al.
Published: (2025)

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
by: Morwani, Depen, et al.
Published: (2025)

A Study on the Calibration of In-context Learning
by: Zhang, Hanlin, et al.
Published: (2023)

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
by: Li, Kenneth, et al.
Published: (2024)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
by: Prabhakar, Akshara, et al.
Published: (2024)

Probing Language Models for Pre-training Data Detection
by: Liu, Zhenhua, et al.
Published: (2024)

Learning Hidden Markov Models Using Conditional Samples
by: Kakade, Sham M., et al.
Published: (2023)

KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
by: Li, Yudong, et al.
Published: (2026)

Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
by: Ali, Mehdi, et al.
Published: (2025)

SOAP: Improving and Stabilizing Shampoo using Adam
by: Vyas, Nikhil, et al.
Published: (2024)

An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
by: Labrak, Yanis, et al.
Published: (2025)

PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
by: Ranaldi, Leonardo, et al.
Published: (2023)

Efficient Evaluation of Large Language Models via Collaborative Filtering
by: Zhong, Xu-Xiang, et al.
Published: (2025)

Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System
by: Muhaimin, Sheikh Samit, et al.
Published: (2025)

LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning
by: Li, Jiachun, et al.
Published: (2024)

Can Pre-trained Language Models Understand Chinese Humor?
by: Chen, Yuyan, et al.
Published: (2024)

Pre-trained Large Language Models for Financial Sentiment Analysis
by: Luo, Wei, et al.
Published: (2024)

Spike No More: Stabilizing the Pre-training of Large Language Models
by: Takase, Sho, et al.
Published: (2023)

Universal Length Generalization with Turing Programs
by: Hou, Kaiying, et al.
Published: (2024)

Cognitive models can reveal interpretable value trade-offs in language models
by: Murthy, Sonia K., et al.
Published: (2025)

From N-grams to Pre-trained Multilingual Models For Language Identification
by: Sindane, Thapelo, et al.
Published: (2024)

Understanding Data Temporality Impact on Large Language Models Pre-training
by: Pilchen, Hippolyte, et al.
Published: (2026)

Boosting Explainability through Selective Rationalization in Pre-trained Language Models
by: Yuan, Libing, et al.
Published: (2025)

RegMix: Data Mixture as Regression for Language Model Pre-training
by: Liu, Qian, et al.
Published: (2024)

DataMan: Data Manager for Pre-training Large Language Models
by: Peng, Ru, et al.
Published: (2025)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)

Aligning Pre-trained Models for Spoken Language Translation
by: Sedláček, Šimon, et al.
Published: (2024)

Sequence-to-Sequence Spanish Pre-trained Language Models
by: Araujo, Vladimir, et al.
Published: (2023)