Image de couverture de livre

Does your data spark joy? Performance gains from domain upsampling at the end of training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Blakeney, Cody, Paul, Mansheej, Larsen, Brett W., Owen, Sean, Frankle, Jonathan
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
par: Ankner, Zachary, et autres
Publié: (2024)

LoRA Learns Less and Forgets Less
par: Biderman, Dan, et autres
Publié: (2024)

Soup to go: mitigating forgetting during continual learning with model averaging
par: Kleiman, Anat, et autres
Publié: (2025)

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
par: Sardana, Nikhil, et autres
Publié: (2023)

Scaling Laws for Precision
par: Kumar, Tanishq, et autres
Publié: (2024)

Continual Pre-training of MoEs: How robust is your router?
par: Thérien, Benjamin, et autres
Publié: (2025)

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
par: Huang, Chengyu, et autres
Publié: (2026)

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
par: Portes, Jacob, et autres
Publié: (2023)

Variance Control via Weight Rescaling in LLM Pre-training
par: Owen, Louis, et autres
Publié: (2025)

ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography
par: Ahrens, Lara, et autres
Publié: (2025)

Critique-out-Loud Reward Models
par: Ankner, Zachary, et autres
Publié: (2024)

$μ$nit Scaling: Simple and Scalable FP8 LLM Training
par: Narayan, Saaketh, et autres
Publié: (2025)

Don't lie to your friends: Learning what you know from collaborative self-play
par: Eisenstein, Jacob, et autres
Publié: (2025)

Luxical: High-Speed Lexical-Dense Text Embeddings
par: DatologyAI, et autres
Publié: (2025)

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
par: DatologyAI, et autres
Publié: (2025)

Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator
par: Bataev, Vladimir, et autres
Publié: (2023)

Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models
par: Brett, Marshall
Publié: (2026)

Does Prompt Formatting Have Any Impact on LLM Performance?
par: He, Jia, et autres
Publié: (2024)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
par: Thangarasa, Vithursan, et autres
Publié: (2024)

How Does Code Pretraining Affect Language Model Task Performance?
par: Petty, Jackson, et autres
Publié: (2024)

Data-Prep-Kit: getting your data ready for LLM application development
par: Wood, David, et autres
Publié: (2024)

LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
par: Łazuka, Małgorzata, et autres
Publié: (2024)

Replaying pre-training data improves fine-tuning
par: Kotha, Suhas, et autres
Publié: (2026)

A State-of-the-Art SQL Reasoning Model using RLVR
par: Ali, Alnur, et autres
Publié: (2025)

Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective
par: Bai, Xueying, et autres
Publié: (2024)

Does Biomedical Training Lead to Better Medical Performance?
par: Dada, Amin, et autres
Publié: (2024)

Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs
par: Ranjani, HG, et autres
Publié: (2025)

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?
par: Sakai, Yusuke, et autres
Publié: (2023)

Explorations of Self-Repair in Language Models
par: Rushing, Cody, et autres
Publié: (2024)

End-to-end Planner Training for Language Modeling
par: Cornille, Nathan, et autres
Publié: (2024)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
par: Liu, Lei, et autres
Publié: (2025)

Does Overnight News Explain Overnight Returns?
par: Glasserman, Paul, et autres
Publié: (2025)

Spiking the training data to correct for test set contamination
par: Wei, Johnny Tian-Zheng, et autres
Publié: (2026)

Non-Determinism and the Lawlessness of Machine Learning Code
par: Cooper, A. Feder, et autres
Publié: (2022)

Using Pre-trained LLMs for Multivariate Time Series Forecasting
par: Wolff, Malcolm L., et autres
Publié: (2025)

An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models
par: Shyr, Cathy, et autres
Publié: (2026)

A comparison of data filtering techniques for English-Polish LLM-based machine translation in the biomedical domain
par: Lérida, Jorge del Pozo, et autres
Publié: (2025)

CELL your Model: Contrastive Explanations for Large Language Models
par: Luss, Ronny, et autres
Publié: (2024)

Unified Latents (UL): How to train your latents
par: Heek, Jonathan, et autres
Publié: (2026)

Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
par: Chen, Peter, et autres
Publié: (2025)