Image de couverture de livre

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Soldaini, Luca, Kinney, Rodney, Bhagia, Akshita, Schwenk, Dustin, Atkinson, David, Authur, Russell, Bogin, Ben, Chandu, Khyathi, Dumas, Jennifer, Elazar, Yanai, Hofmann, Valentin, Jha, Ananya Harsh, Kumar, Sachin, Lucy, Li, Lyu, Xinxi, Lambert, Nathan, Magnusson, Ian, Morrison, Jacob, Muennighoff, Niklas, Naik, Aakanksha, Nam, Crystal, Peters, Matthew E., Ravichander, Abhilasha, Richardson, Kyle, Shen, Zejiang, Strubell, Emma, Subramani, Nishant, Tafjord, Oyvind, Walsh, Pete, Zettlemoyer, Luke, Smith, Noah A., Hajishirzi, Hannaneh, Beltagy, Iz, Groeneveld, Dirk, Dodge, Jesse, Lo, Kyle
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Paloma: A Benchmark for Evaluating Language Model Fit
par: Magnusson, Ian, et autres
Publié: (2023)

OLMo: Accelerating the Science of Language Models
par: Groeneveld, Dirk, et autres
Publié: (2024)

What's In My Big Data?
par: Elazar, Yanai, et autres
Publié: (2023)

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
par: Liu, Jiacheng, et autres
Publié: (2024)

RESTOR: Knowledge Recovery in Machine Unlearning
par: Rezaei, Keivan, et autres
Publié: (2024)

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
par: Wettig, Alexander, et autres
Publié: (2025)

Establishing Task Scaling Laws via Compute-Efficient Model Ladders
par: Bhagia, Akshita, et autres
Publié: (2024)

Agent Lumos: Unified and Modular Training for Open-Source Language Agents
par: Yin, Da, et autres
Publié: (2023)

HREF: Human Response-Guided Evaluation of Instruction Following in Language Models
par: Lyu, Xinxi, et autres
Publié: (2024)

The Art of Saying No: Contextual Noncompliance in Language Models
par: Brahman, Faeze, et autres
Publié: (2024)

SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
par: Wadden, David, et autres
Publié: (2024)

Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
par: Wiegreffe, Sarah, et autres
Publié: (2024)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
par: Magnusson, Ian, et autres
Publié: (2025)

Source-Aware Training Enables Knowledge Attribution in Language Models
par: Khalifa, Muhammad, et autres
Publié: (2024)

Just CHOP: Embarrassingly Simple LLM Compression
par: Jha, Ananya Harsh, et autres
Publié: (2023)

WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
par: Lin, Bill Yuchen, et autres
Publié: (2024)

Olmix: A Framework for Data Mixing Throughout LM Development
par: Chen, Mayee F., et autres
Publié: (2026)

LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv
par: Elazar, Yanai, et autres
Publié: (2026)

FlexOlmo: Open Language Models for Flexible Data Use
par: Shi, Weijia, et autres
Publié: (2025)

OLMoE: Open Mixture-of-Experts Language Models
par: Muennighoff, Niklas, et autres
Publié: (2024)

How Many Images Does It Take? Estimating Imitation Thresholds in Text-to-Image Models
par: Verma, Sahil, et autres
Publié: (2024)

OLMES: A Standard for Language Model Evaluations
par: Gu, Yuling, et autres
Publié: (2024)

Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
par: Chen, Tong, et autres
Publié: (2025)

EMO: Pretraining Mixture of Experts for Emergent Modularity
par: Wang, Ryan, et autres
Publié: (2026)

What Has Been Lost with Synthetic Evaluation?
par: Gill, Alexander, et autres
Publié: (2025)

Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
par: Balepur, Nishant, et autres
Publié: (2024)

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
par: Liu, Jiacheng, et autres
Publié: (2025)

olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
par: Poznanski, Jake, et autres
Publié: (2025)

WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries
par: Zhao, Wenting, et autres
Publié: (2024)

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
par: Miranda, Lester James V., et autres
Publié: (2024)

s1: Simple test-time scaling
par: Muennighoff, Niklas, et autres
Publié: (2025)

olmOCR 2: Unit Test Rewards for Document OCR
par: Poznanski, Jake, et autres
Publié: (2025)

Revisiting the Past: Data Unlearning with Model State History
par: Rezaei, Keivan, et autres
Publié: (2025)

HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
par: Ravichander, Abhilasha, et autres
Publié: (2025)

Evaluating $n$-Gram Novelty of Language Models Using Rusty-DAWG
par: Merrill, William, et autres
Publié: (2024)

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
par: Zhao, Bowen, et autres
Publié: (2024)

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
par: Elazar, Yanai, et autres
Publié: (2023)

RewardBench: Evaluating Reward Models for Language Modeling
par: Lambert, Nathan, et autres
Publié: (2024)

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
par: Min, Sewon, et autres
Publié: (2023)

Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation
par: Iluz, Bar, et autres
Publié: (2024)