Copertina

Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhou, Fan, Wang, Zengzhi, Liu, Qian, Li, Junlong, Liu, Pengfei
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
di: Wang, Zengzhi, et al.
Pubblicazione: (2025)

MathPile: A Billion-Token-Scale Pretraining Corpus for Math
di: Wang, Zengzhi, et al.
Pubblicazione: (2023)

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning
di: Fan, Run-Ze, et al.
Pubblicazione: (2025)

Benchmarking Benchmark Leakage in Large Language Models
di: Xu, Ruijie, et al.
Pubblicazione: (2024)

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
di: Ling Team, et al.
Pubblicazione: (2025)

Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
di: Codefuse, et al.
Pubblicazione: (2025)

CASE: Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models
di: Harne, Sarthak, et al.
Pubblicazione: (2024)

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
di: Nguyen, Thao, et al.
Pubblicazione: (2025)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
di: Yang, Kailai, et al.
Pubblicazione: (2025)

Investigating Data Contamination for Pre-training Language Models
di: Jiang, Minhao, et al.
Pubblicazione: (2024)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
di: Kim, Gyeongman, et al.
Pubblicazione: (2025)

Scaling Embeddings Outperforms Scaling Experts in Language Models
di: Liu, Hong, et al.
Pubblicazione: (2026)

MegaMath: Pushing the Limits of Open Math Corpora
di: Zhou, Fan, et al.
Pubblicazione: (2025)

ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data
di: Liu, Xiaoyang, et al.
Pubblicazione: (2025)

Parallel Structures in Pre-training Data Yield In-Context Learning
di: Chen, Yanda, et al.
Pubblicazione: (2024)

Ask Again, Then Fail: Large Language Models' Vacillations in Judgment
di: Xie, Qiming, et al.
Pubblicazione: (2023)

Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
di: Kang, Feiyang, et al.
Pubblicazione: (2025)

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
di: Gao, Chengqian, et al.
Pubblicazione: (2025)

Reformatted Alignment
di: Fan, Run-Ze, et al.
Pubblicazione: (2024)

RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
di: Bi, Baolong, et al.
Pubblicazione: (2025)

LIMR: Less is More for RL Scaling
di: Li, Xuefeng, et al.
Pubblicazione: (2025)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
di: Wang, Bo, et al.
Pubblicazione: (2026)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
di: Chen, Tong, et al.
Pubblicazione: (2025)

CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model
di: Chiang, Shang-Hsuan, et al.
Pubblicazione: (2024)

From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
di: Tan, Yuqiao, et al.
Pubblicazione: (2026)

Is Programming by Example solved by LLMs?
di: Li, Wen-Ding, et al.
Pubblicazione: (2024)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
di: Bergsma, Shane, et al.
Pubblicazione: (2025)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
di: Samragh, Mohammad, et al.
Pubblicazione: (2024)

HiFloat4 Format for Language Model Pre-training on Ascend NPUs
di: Taghian, Mehran, et al.
Pubblicazione: (2026)

Diversity-oriented Data Augmentation with Large Language Models
di: Wang, Zaitian, et al.
Pubblicazione: (2025)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
di: Kang, Feiyang, et al.
Pubblicazione: (2024)

Reinforcement Learning on Pre-Training Data
di: Li, Siheng, et al.
Pubblicazione: (2025)

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
di: Yi, Rongjie, et al.
Pubblicazione: (2024)

Generating Pragmatic Examples to Train Neural Program Synthesizers
di: Vaduguru, Saujas, et al.
Pubblicazione: (2023)

DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data
di: Zhou, Yuhang, et al.
Pubblicazione: (2025)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
di: Tang, Shengkun, et al.
Pubblicazione: (2026)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
di: Ling Team, et al.
Pubblicazione: (2025)

MobileMoE: Scaling On-Device Mixture of Experts
di: Chen, Yanbei, et al.
Pubblicazione: (2026)

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data
di: Bogdanov, Sergei, et al.
Pubblicazione: (2024)