Cover Image

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Na, Clara, Magnusson, Ian, Jha, Ananya Harsh, Sherborne, Tom, Strubell, Emma, Dodge, Jesse, Dasigi, Pradeep
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Just CHOP: Embarrassingly Simple LLM Compression
by: Jha, Ananya Harsh, et al.
Published: (2023)

TRAM: Bridging Trust Regions and Sharpness Aware Minimization
by: Sherborne, Tom, et al.
Published: (2023)

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
by: Morrison, Jacob, et al.
Published: (2024)

Holistically Evaluating the Environmental Impact of Creating Language Models
by: Morrison, Jacob, et al.
Published: (2025)

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
by: Lucy, Li, et al.
Published: (2024)

To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing
by: Gururaja, Sireesh, et al.
Published: (2023)

Energy Considerations of Large Language Model Inference and Efficiency Optimizations
by: Fernandez, Jared, et al.
Published: (2025)

Energy and Carbon Considerations of Fine-Tuning BERT
by: Wang, Xiaorong, et al.
Published: (2023)

HREF: Human Response-Guided Evaluation of Instruction Following in Language Models
by: Lyu, Xinxi, et al.
Published: (2024)

Gradient Localization Improves Lifelong Pretraining of Language Models
by: Fernandez, Jared, et al.
Published: (2024)

Paloma: A Benchmark for Evaluating Language Model Fit
by: Magnusson, Ian, et al.
Published: (2023)

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
by: Heineman, David, et al.
Published: (2025)

Large-Scale Data Selection for Instruction Tuning
by: Ivison, Hamish, et al.
Published: (2025)

Evaluating In-Context Learning of Libraries for Code Generation
by: Patel, Arkil, et al.
Published: (2023)

Compositional Generalisation for Explainable Hate Speech Detection
by: Calabrese, Agostina, et al.
Published: (2025)

If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs
by: Khalifa, Muhammad, et al.
Published: (2024)

OLMo: Accelerating the Science of Language Models
by: Groeneveld, Dirk, et al.
Published: (2024)

Source-Aware Training Enables Knowledge Attribution in Language Models
by: Khalifa, Muhammad, et al.
Published: (2024)

Fluid Language Model Benchmarking
by: Hofmann, Valentin, et al.
Published: (2025)

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
by: Soldaini, Luca, et al.
Published: (2024)

Establishing Task Scaling Laws via Compute-Efficient Model Ladders
by: Bhagia, Akshita, et al.
Published: (2024)

Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
by: Engländer, Leon, et al.
Published: (2026)

What's In My Big Data?
by: Elazar, Yanai, et al.
Published: (2023)

Language Models Hallucinate, but May Excel at Fact Verification
by: Guan, Jian, et al.
Published: (2023)

Beyond Text: Characterizing Domain Expert Needs in Document Research
by: Gururaja, Sireesh, et al.
Published: (2025)

FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction
by: Johnson, Natasha, et al.
Published: (2025)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
by: Magnusson, Ian, et al.
Published: (2025)

SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
by: Kundurthy, Srivatsa, et al.
Published: (2026)

Stereotype or Personalization? User Identity Biases Chatbot Recommendations
by: Kantharuban, Anjali, et al.
Published: (2024)

Approximating Language Model Training Data from Weights
by: Morris, John X., et al.
Published: (2025)

Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
by: Lu, Zhenyi, et al.
Published: (2024)

OLMES: A Standard for Language Model Evaluations
by: Gu, Yuling, et al.
Published: (2024)

Generalizing Verifiable Instruction Following
by: Pyatkin, Valentina, et al.
Published: (2025)

Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models
by: Ding, Zhuojun, et al.
Published: (2025)

ReFIT: Relevance Feedback from a Reranker during Inference
by: Reddy, Revanth Gangi, et al.
Published: (2023)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
by: Sawada, Tomohiro, et al.
Published: (2025)

ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans
by: Sadana, Ananya, et al.
Published: (2025)

Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
by: Subramanian, Seganrasan, et al.
Published: (2025)

Kinetics: Rethinking Test-Time Scaling Laws
by: Sadhukhan, Ranajoy, et al.
Published: (2025)

Scalable Multi-Domain Adaptation of Language Models using Modular Experts
by: Schafhalter, Peter, et al.
Published: (2024)