Copertina

Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wang, Ruizhe, Ding, Yucheng, Liu, Xiao, Wang, Yaoxiang, Cheng, Peng, Guo, Baining, Zha, Zhengjun, Gong, Yeyun
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
di: Wang, Yaoxiang, et al.
Pubblicazione: (2025)

Optimizing Large Language Model Training Using FP4 Quantization
di: Wang, Ruizhe, et al.
Pubblicazione: (2025)

Mixture of Neuron Experts
di: Cheng, Runxi, et al.
Pubblicazione: (2025)

m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder
di: Wang, Yaoxiang, et al.
Pubblicazione: (2026)

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
di: Ren, Lianhai, et al.
Pubblicazione: (2026)

Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
di: Luo, Zheheng, et al.
Pubblicazione: (2024)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
di: Liu, Boan, et al.
Pubblicazione: (2023)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
di: Yang, Kailai, et al.
Pubblicazione: (2025)

Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
di: Wang, Qi, et al.
Pubblicazione: (2025)

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
di: Wang, Hong, et al.
Pubblicazione: (2025)

Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
di: Zhan, Zheng, et al.
Pubblicazione: (2025)

Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM
di: Zhang, Fengrun, et al.
Pubblicazione: (2024)

Progress Decisions Involving Time: Sunk Cost or Completion Effects
di: Todd J. Thorsteinson, et al.
Pubblicazione: (2024)

Loss Aversion as a Potential Factor in the Sunk-Cost Fallacy
di: Veronika Tait
Pubblicazione: (2019)

Sunk Costs: does group decision make a difference?
di: Ana Luiza Paraboni
Pubblicazione: (2019)

PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization
di: Ma, Xinbei, et al.
Pubblicazione: (2023)

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
di: Liu, Jiashun, et al.
Pubblicazione: (2025)

Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning
di: Wang, Huiyi, et al.
Pubblicazione: (2024)

PEPT: Expert Finding Meets Personalized Pre-training
di: Peng, Qiyao, et al.
Pubblicazione: (2023)

Generic Knowledge Boosted Pre-training For Remote Sensing Images
di: Huang, Ziyue, et al.
Pubblicazione: (2024)

Facet-Aware Multi-Head Mixture-of-Experts Model with Text-Enhanced Pre-training for Sequential Recommendation
di: Liu, Mingrui, et al.
Pubblicazione: (2026)

Mixture to Beamformed Mixture: Leveraging Beamformed Mixture as Weak-Supervision for Speech Enhancement and Noise-Robust ASR
di: Wang, Zhong-Qiu, et al.
Pubblicazione: (2025)

Efeito Sunk Costs: O Conhecimento Teórico Influencia no Processo Decisório de Discentes?
di: Suliani Rover
Pubblicazione: (2009)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
di: Zhong, Zexuan, et al.
Pubblicazione: (2024)

Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
di: Deng, Haoran, et al.
Pubblicazione: (2025)

Incorporating Pre-trained Diffusion Models in Solving the Schrödinger Bridge Problem
di: Tang, Zhicong, et al.
Pubblicazione: (2025)

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
di: Shin, Haebin, et al.
Pubblicazione: (2025)

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
di: Zhu, Tong, et al.
Pubblicazione: (2024)

Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
di: Hendawy, Ahmed, et al.
Pubblicazione: (2023)

Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving
di: Choi, Junsun, et al.
Pubblicazione: (2026)

Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
di: Cheng, Runxi, et al.
Pubblicazione: (2026)

Boosting 3D Neuron Segmentation with 2D Vision Transformer Pre-trained on Natural Images
di: Cheng, Yik San, et al.
Pubblicazione: (2024)

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
di: Lin, Xi Victoria, et al.
Pubblicazione: (2024)

Sigma-MoE-Tiny Technical Report
di: Hu, Qingguo, et al.
Pubblicazione: (2025)

SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
di: Qu, Lei, et al.
Pubblicazione: (2025)

Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony
di: Wang, Shaoyu, et al.
Pubblicazione: (2025)

Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions
di: Duan, Boyan, et al.
Pubblicazione: (2025)

Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy
di: Yang, Yiting, et al.
Pubblicazione: (2025)

SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency
di: Kyung, Kwanhee, et al.
Pubblicazione: (2025)

Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
di: Li, Haoling, et al.
Pubblicazione: (2024)