Buchumschlag

Scalable Training of Mixture-of-Experts Models with Megatron Core

Fuente: arXiv

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Yan, Zijie, Bai, Hongxiao, Yao, Xin, Liu, Dennis, Liu, Tong, Liu, Hongbin, Li, Pingtian, Wu, Evan, Fan, Shiqing, Tao, Li, Zhang, Robin, Wang, Yuzhong, Xu, Shifang, Chang, Jack, Chen, Xuwen, Li, Kunlun, Bai, Yan, Deng, Gao, Zheng, Nan, Korthikanti, Vijay Anand, Khattar, Abhinav, He, Ethan, Govande, Soham, Lym, Sangkug, Zhu, Zhongbo, Zhang, Qi, Yuan, Haochen, Ren, Xiaowei, Fu, Deyu, Ma, Tailai, Zhang, Shunkang, Shao, Jiang, Wang, Ray, Rengasamy, Vasudevan, Garg, Rachit, Bhavani, Santosh, Li, Xipeng, Zhou, Chandler, Wu, David, Wei, Yingcan, Aithal, Ashwath, Andersch, Michael, Shoeybi, Mohammad, Yao, Jiajie, Yang, June
Format:	Preprint
Veröffentlicht:	2026
Schlagworte:	Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning
Online-Zugang:	Acceder al recurso
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Ähnliche Einträge

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
von: Liu, Dennis, et al.
Veröffentlicht: (2025)

Upcycling Large Language Models into Mixture of Experts
von: He, Ethan, et al.
Veröffentlicht: (2024)

DNA Cleavage System by Nanomaterials
von: Jinci Li, et al.
Veröffentlicht: (2024)

MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training
von: Zhao, Bohan, et al.
Veröffentlicht: (2025)

Megatron: Evasive Clean-Label Backdoor Attacks against Vision Transformer
von: Gong, Xueluan, et al.
Veröffentlicht: (2024)

An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
von: Govande, Soham V.
Veröffentlicht: (2024)

Efficient GPU-Centered Singular Value Decomposition Using the Divide-and-Conquer Method
von: Liu, Shifang, et al.
Veröffentlicht: (2025)

Heterogeneous Parallelism for Multimodal Large Language Model Training
von: Karnati, Yashaswi, et al.
Veröffentlicht: (2026)

Knockdown of LOC100911723 Attenuates Hypoxia‐Induced Autophagy and Apoptosis of Myocardial Cells in Myocardial Infarction
von: Zhengbing Liu, et al.
Veröffentlicht: (2026)

Llama 3 Meets MoE: Efficient Upcycling
von: Vavre, Aditya, et al.
Veröffentlicht: (2024)

Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptation
von: Li, Anqi, et al.
Veröffentlicht: (2024)

Ligament Inspired Ultra‐Strong and Tough Bio‐Based Polyurethane Elastomers via Dynamic Hydrogen Bonding Induced Confinement Effect
von: Yuting Chu, et al.
Veröffentlicht: (2025)

Cooperative colorings of hypergraphs
von: Bai, Xuqing, et al.
Veröffentlicht: (2024)

Qualitative and Comparative Study of APORT Apple Peel Based on UPLC‐Q‐TOF‐MS Technology and Fruit Peel Comparison of Different Provenances
von: Xinyue Dong, et al.
Veröffentlicht: (2025)

Disturbed nutrient accumulation and cell wall metabolism in panicles are responsible for rice straighthead disease
von: Qinghui Liu, et al.
Veröffentlicht: (2024)

A Novel Convolutional Neural Network Architecture with a Continuous Symmetry
von: Liu, Yao, et al.
Veröffentlicht: (2023)

Narrowband Macrocycles: A New Paradigm for High‐Color‐Purity Organic Light‐Emitting Diodes
von: Zhongbo Xiang, et al.
Veröffentlicht: (2026)

ThreatIntel-Andro: Expert-Verified Benchmarking for Robust Android Malware Research
von: Bai, Hongpeng, et al.
Veröffentlicht: (2025)

Isolated zero-energy flat-bands and intrinsic magnetism in carbon monolayers
von: He, Chaoyu, et al.
Veröffentlicht: (2024)

Stacking-Engineered Switchable Altermagnetism in Topological FeSe bilayer systems
von: Li, Jie, et al.
Veröffentlicht: (2026)

Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning
von: Liu, Man, et al.
Veröffentlicht: (2024)

Cycle Pixel Difference Network for Crisp Edge Detection
von: Liu, Changsong, et al.
Veröffentlicht: (2024)

Femtosecond Laser Inscribed Multiwavelength Cascaded FBG Array Using Point‐By‐Point Method for Temperature Sensing Based on Swept Laser Demodulation Technique
von: Jia Liu, et al.
Veröffentlicht: (2025)

Genomic and functional insights into dietary diversification in New World leaf‐nosed bats (Phyllostomidae)
von: Yiran Xu, et al.
Veröffentlicht: (2024)

AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignment
von: Lin, Yiheng, et al.
Veröffentlicht: (2025)

Dual Reasoning: A GNN-LLM Collaborative Framework for Knowledge Graph Question Answering
von: Liu, Guangyi, et al.
Veröffentlicht: (2024)

Constrained Online Recursive Source Separation Framework for Real-time Electrophysiological Signal Processing
von: Li, Yao, et al.
Veröffentlicht: (2024)

Association of Visceral Obesity Indices With Survival Among Cancer Survivors: Evidence From Two Nationally Population‐Based Cohort Study
von: Qiang Li, et al.
Veröffentlicht: (2026)

An Analytical Seismic Response for Rigid Underground Rectangular Tunnel Subjected to Incident P Waves
von: Yang Liu, et al.
Veröffentlicht: (2024)

High‐temperature and high‐pressure dyeing of polyester fabric with sappan wood extract: Colour characteristics and ultraviolet resistance
von: Ke Li, et al.
Veröffentlicht: (2025)

A Dual-Path Framework with Frequency-and-Time Excited Network for Anomalous Sound Detection
von: Zhang, Yucong, et al.
Veröffentlicht: (2024)

Astragaloside IV Attenuates Myocardial Ischemia‐Reperfusion Injury via Suppression of the NOX4/ROS Axis: Mechanistic Insights From In Vivo and In Vitro Models
von: Xiao Liu, et al.
Veröffentlicht: (2026)

The Effect of Current Intensity on Macrosegregation in Vacuum Arc Remelting Cr14Mo4V Ingot Based on a Three‐Phase Mixed Columnar‐Equiaxed Model
von: Long Zhao, et al.
Veröffentlicht: (2026)

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention
von: Liu, Ying, et al.
Veröffentlicht: (2024)

TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
von: Liu, Yuliang, et al.
Veröffentlicht: (2024)

Decomposing and Steering Functional Metacognition in Large Language Models
von: Li, Yanshi, et al.
Veröffentlicht: (2026)

CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems
von: Tang, Yongxiang, et al.
Veröffentlicht: (2022)

MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling
von: Yin, Liang, et al.
Veröffentlicht: (2025)

TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
von: Bai, Sikai, et al.
Veröffentlicht: (2026)

An alignment algorithm using coherent twin boundaries as internal reference in 3D‐EBSD
von: Heng Li, et al.
Veröffentlicht: (2024)