Copertina

BiMix: A Bivariate Data Mixing Law for Language Model Pretraining

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Ge, Ce, Ma, Zhijian, Chen, Daoyuan, Li, Yaliang, Ding, Bolin
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Provable Scaling Laws for the Test-Time Compute of Large Language Models
di: Chen, Yanxi, et al.
Pubblicazione: (2024)

Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
di: Chen, Yanxi, et al.
Pubblicazione: (2024)

Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development
di: Chen, Daoyuan, et al.
Pubblicazione: (2024)

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
di: Pan, Xuchen, et al.
Pubblicazione: (2024)

Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data
di: Ling, Zhenqing, et al.
Pubblicazione: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
di: Ye, Jiasheng, et al.
Pubblicazione: (2024)

Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
di: He, Shiqi, et al.
Pubblicazione: (2025)

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
di: Qin, Zhen, et al.
Pubblicazione: (2024)

Rethinking Data Mixing from the Perspective of Large Language Models
di: Xu, Yuanjian, et al.
Pubblicazione: (2026)

Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
di: Wei, Fei, et al.
Pubblicazione: (2025)

Aioli: A Unified Optimization Framework for Language Model Data Mixing
di: Chen, Mayee F., et al.
Pubblicazione: (2024)

Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
di: Yao, Chaorui, et al.
Pubblicazione: (2025)

MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
di: Zhang, Shiyue, et al.
Pubblicazione: (2023)

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
di: Yang, Jinluan, et al.
Pubblicazione: (2025)

Merge to Mix: Mixing Datasets via Model Merging
di: Tao, Zhixu Silvia, et al.
Pubblicazione: (2025)

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
di: Hu, Michael Y., et al.
Pubblicazione: (2026)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
di: Foroutan, Negar, et al.
Pubblicazione: (2025)

The Impact of Language Mixing on Bilingual LLM Reasoning
di: Li, Yihao, et al.
Pubblicazione: (2025)

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
di: Chen, Yanxi, et al.
Pubblicazione: (2023)

Towards Anthropomorphic Conversational AI Part I: A Practical Framework
di: Wei, Fei, et al.
Pubblicazione: (2025)

Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
di: Gu, Xinran, et al.
Pubblicazione: (2025)

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
di: Bai, Jiamu, et al.
Pubblicazione: (2024)

Olmix: A Framework for Data Mixing Throughout LM Development
di: Chen, Mayee F., et al.
Pubblicazione: (2026)

Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
di: Zhuo, Zhijian, et al.
Pubblicazione: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
di: Kang, Feiyang, et al.
Pubblicazione: (2024)

Can Language Models Discover Scaling Laws?
di: Lin, Haowei, et al.
Pubblicazione: (2025)

Discovering Knowledge-Critical Subnetworks in Pretrained Language Models
di: Bayazit, Deniz, et al.
Pubblicazione: (2023)

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
di: Wang, Xinyi, et al.
Pubblicazione: (2024)

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
di: Singh, Karan, et al.
Pubblicazione: (2026)

Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
di: Ali, Mehdi, et al.
Pubblicazione: (2025)

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
di: Ling, Zhenqing, et al.
Pubblicazione: (2024)

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
di: Guan, Ziyi, et al.
Pubblicazione: (2024)

Fine-tuning can Help Detect Pretraining Data from Large Language Models
di: Zhang, Hengxiang, et al.
Pubblicazione: (2024)

In-context Pretraining: Language Modeling Beyond Document Boundaries
di: Shi, Weijia, et al.
Pubblicazione: (2023)

Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models
di: Jiao, Qirui, et al.
Pubblicazione: (2024)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
di: Yang, Kailai, et al.
Pubblicazione: (2025)

MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
di: Chen, Zhixun, et al.
Pubblicazione: (2025)

Pretraining Large Language Models with NVFP4
di: NVIDIA, et al.
Pubblicazione: (2025)

Patent Language Model Pretraining with ModernBERT
di: Yousefiramandi, Amirhossein, et al.
Pubblicazione: (2025)

Towards the Law of Capacity Gap in Distilling Language Models
di: Zhang, Chen, et al.
Pubblicazione: (2023)