Copertina

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Lv, Xingtai, Sheng, Li, Zhang, Kaiyan, You, Yichen, Gao, Siyan, Luo, Xueheng, Zuo, Yuxin, Fan, Yuchen, Yang, Junlin, Cui, Ganqu, Wang, Bingning, Yang, Fan, Sun, Youbang, Ding, Ning, Zhou, Bowen
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Towards a Unified View of Large Language Model Post-Training
di: Lv, Xingtai, et al.
Pubblicazione: (2025)

How Far Can Unsupervised RLVR Scale LLM Training?
di: He, Bingxiang, et al.
Pubblicazione: (2026)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
di: Lv, Xingtai, et al.
Pubblicazione: (2024)

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
di: Hua, Ermo, et al.
Pubblicazione: (2024)

Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
di: Lv, Xingtai, et al.
Pubblicazione: (2025)

OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
di: Wang, Liujianfu, et al.
Pubblicazione: (2025)

SD-MoE: Spectral Decomposition for Effective Expert Specialization
di: Huang, Ruijun, et al.
Pubblicazione: (2026)

MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
di: Zhang, Geng, et al.
Pubblicazione: (2025)

Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
di: Li, Pingzhi, et al.
Pubblicazione: (2025)

MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
di: Pang, Yuqi, et al.
Pubblicazione: (2025)

Automating Exploratory Proteomics Research via Language Models
di: Ding, Ning, et al.
Pubblicazione: (2024)

Automating Exploratory Multiomics Research via Language Models
di: Qu, Shang, et al.
Pubblicazione: (2025)

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
di: Shi, Chufan, et al.
Pubblicazione: (2024)

MoE Pathfinder: Trajectory-driven Expert Pruning
di: Yang, Xican, et al.
Pubblicazione: (2025)

DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
di: Feng, Yuchen, et al.
Pubblicazione: (2025)

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
di: Zhu, Peijun, et al.
Pubblicazione: (2025)

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
di: Zhang, Yue, et al.
Pubblicazione: (2025)

Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
di: Cong, Peizhuang, et al.
Pubblicazione: (2024)

TTRL: Test-Time Reinforcement Learning
di: Zuo, Yuxin, et al.
Pubblicazione: (2025)

SkipGS: Post-Densification Backward Skipping for Efficient 3DGS Training
di: Li, Jingxing, et al.
Pubblicazione: (2026)

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
di: Li, Haozhan, et al.
Pubblicazione: (2025)

Advancing Expert Specialization for Better MoE
di: Guo, Hongcan, et al.
Pubblicazione: (2025)

TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts
di: Xu, Yu, et al.
Pubblicazione: (2026)

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
di: Zhao, Ziyu, et al.
Pubblicazione: (2026)

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
di: Zhang, Kaiyan, et al.
Pubblicazione: (2024)

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
di: Hua, Ermo, et al.
Pubblicazione: (2024)

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
di: Li, Zichong, et al.
Pubblicazione: (2025)

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
di: Ludziejewski, Jan, et al.
Pubblicazione: (2025)

FlowRL: Matching Reward Distributions for LLM Reasoning
di: Zhu, Xuekai, et al.
Pubblicazione: (2025)

BioFact-MoE: Biologically Factorized Mixture of Experts for Vision-Language Prognostic Modeling in Hepatocellular Carcinoma
di: Yang, Junlin, et al.
Pubblicazione: (2026)

ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
di: Xu, Yuhao, et al.
Pubblicazione: (2026)

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
di: Wu, Haoyuan, et al.
Pubblicazione: (2025)

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
di: Qian, Yulei, et al.
Pubblicazione: (2024)

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
di: Liu, Yang, et al.
Pubblicazione: (2026)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
di: Pan, Xinglin, et al.
Pubblicazione: (2025)

PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
di: Su, Yang, et al.
Pubblicazione: (2025)

Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference
di: Sun, Xun, et al.
Pubblicazione: (2026)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
di: Gu, Naibin, et al.
Pubblicazione: (2025)

BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
di: Wu, Juntong, et al.
Pubblicazione: (2026)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
di: Takashiro, Shota, et al.
Pubblicazione: (2026)