Image de couverture de livre

DataComp-LM: In search of the next generation of training sets for language models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Li, Jeffrey, Fang, Alex, Smyrnis, Georgios, Ivgi, Maor, Jordan, Matt, Gadre, Samir, Bansal, Hritik, Guha, Etash, Keh, Sedrick, Arora, Kushal, Garg, Saurabh, Xin, Rui, Muennighoff, Niklas, Heckel, Reinhard, Mercat, Jean, Chen, Mayee, Gururangan, Suchin, Wortsman, Mitchell, Albalak, Alon, Bitton, Yonatan, Nezhurina, Marianna, Abbas, Amro, Hsieh, Cheng-Yu, Ghosh, Dhruba, Gardner, Josh, Kilian, Maciej, Zhang, Hanlin, Shao, Rulin, Pratt, Sarah, Sanyal, Sunny, Ilharco, Gabriel, Daras, Giannis, Marathe, Kalyani, Gokaslan, Aaron, Zhang, Jieyu, Chandu, Khyathi, Nguyen, Thao, Vasiljevic, Igor, Kakade, Sham, Song, Shuran, Sanghavi, Sujay, Faghri, Fartash, Oh, Sewoong, Zettlemoyer, Luke, Lo, Kyle, El-Nouby, Alaaeldin, Pouransari, Hadi, Toshev, Alexander, Wang, Stephanie, Groeneveld, Dirk, Soldaini, Luca, Koh, Pang Wei, Jitsev, Jenia, Kollar, Thomas, Dimakis, Alexandros G., Carmon, Yair, Dave, Achal, Schmidt, Ludwig, Shankar, Vaishaal
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Language models scale reliably with over-training and on downstream tasks
par: Gadre, Samir Yitzhak, et autres
Publié: (2024)

MobileCLIP2: Improving Multi-Modal Reinforced Training
par: Faghri, Fartash, et autres
Publié: (2025)

TiC-CLIP: Continual Training of CLIP Models
par: Garg, Saurabh, et autres
Publié: (2023)

Scalable Pre-training of Large Autoregressive Image Models
par: El-Nouby, Alaaeldin, et autres
Publié: (2024)

CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2024)

Resolving Discrepancies in Compute-Optimal Scaling of Language Models
par: Porian, Tomer, et autres
Publié: (2024)

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2023)

Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
par: Fang, Alex, et autres
Publié: (2025)

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
par: Li, Jeffrey, et autres
Publié: (2025)

Linearizing Large Language Models
par: Mercat, Jean, et autres
Publié: (2024)

Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
par: Vemulapalli, Raviteja, et autres
Publié: (2023)

Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
par: Cipolina-Kun, Lucia, et autres
Publié: (2025)

Accelerated Parameter-Free Stochastic Optimization
par: Kreisler, Itai, et autres
Publié: (2024)

Time is Encoded in the Weights of Finetuned Language Models
par: Nylund, Kai, et autres
Publié: (2023)

Reproducible scaling laws for contrastive language-image learning
par: Cherti, Mehdi, et autres
Publié: (2022)

MUSCLE: A Model Update Strategy for Compatible LLM Evolution
par: Echterhoff, Jessica, et autres
Publié: (2024)

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
par: Nezhurina, Marianna, et autres
Publié: (2024)

LESS: Selecting Influential Data for Targeted Instruction Tuning
par: Xia, Mengzhou, et autres
Publié: (2024)

Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
par: Huang, Chen, et autres
Publié: (2025)

SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
par: Wang, Haoxiang, et autres
Publié: (2023)

FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
par: Hsieh, Cheng-Yu, et autres
Publié: (2025)

Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
par: Zhang, Yuhui, et autres
Publié: (2023)

VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2026)

Should VLMs be Pre-trained with Image Data?
par: Keh, Sedrick, et autres
Publié: (2025)

Exploring and Applying Audio-Based Sentiment Analysis in Music
par: Jhanji, Etash
Publié: (2024)

Scaling Laws for Optimal Data Mixtures
par: Shukor, Mustafa, et autres
Publié: (2025)

Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
par: Nezhurina, Marianna, et autres
Publié: (2025)

Learning in Compact Spaces with Approximately Normalized Transformer
par: Franke, Jörg K. H., et autres
Publié: (2025)

OpenThoughts: Data Recipes for Reasoning Models
par: Guha, Etash, et autres
Publié: (2025)

FastVLM: Efficient Vision Encoding for Vision Language Models
par: Vasu, Pavan Kumar Anasosalu, et autres
Publié: (2024)

Scaling Laws for Native Multimodal Models
par: Shukor, Mustafa, et autres
Publié: (2025)

Coboundaries and eigenvalues of morphic subshifts
par: Mercat, Paul
Publié: (2024)

Geometrical representation of subshifts for primitive substitutions
par: Mercat, Paul
Publié: (2022)

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
par: Min, Sewon, et autres
Publié: (2023)

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
par: Blevins, Terra, et autres
Publié: (2024)

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
par: Lucy, Li, et autres
Publié: (2024)

Solving Robust MDPs through No-Regret Dynamics
par: Guha, Etash Kumar
Publié: (2023)

Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison
par: Nezhurina, Marianna, et autres
Publié: (2025)

VLA Foundry: A Unified Framework for Training Vision-Language-Action Models
par: Mercat, Jean, et autres
Publié: (2026)

Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
par: Jayalath, Dulhan, et autres
Publié: (2025)