Copertina

Mixture-of-Channels: Exploiting Sparse FFNs for Efficient LLMs Pre-Training and Inference

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wu, Tong, He, Yutong, Wang, Bin, Yuan, Kun
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

PreMoE: Proactive Inference for Efficient Mixture-of-Experts
di: Pei, Zehua, et al.
Pubblicazione: (2025)

MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
di: Liu, Yuxi, et al.
Pubblicazione: (2025)

CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
di: Chen, Guanduo, et al.
Pubblicazione: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers
di: Liu, Yuxi, et al.
Pubblicazione: (2026)

S2Aligner: Pair-Efficient and Transferable Pre-Training for Sparse Text-Attributed Graphs
di: Wang, Yuhan, et al.
Pubblicazione: (2026)

RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
di: Liu, Yuxi, et al.
Pubblicazione: (2026)

Topology-Aware Revival for Efficient Sparse Training
di: Jin, Meiling, et al.
Pubblicazione: (2026)

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
di: Liu, Enshu, et al.
Pubblicazione: (2024)

ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs
di: Butler, Landon, et al.
Pubblicazione: (2025)

SparseDVFS: Sparse-Aware DVFS for Energy-Efficient Edge Inference
di: Zhang, Ziyang, et al.
Pubblicazione: (2026)

Gate the Filter, Not the Message: Node-Channel Mixtures for Pre-Propagation GNNs
di: Yue, Zichao, et al.
Pubblicazione: (2026)

BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
di: Wang, Yun, et al.
Pubblicazione: (2025)

Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
di: Li, Zhexiang, et al.
Pubblicazione: (2025)

Efficient Pre-Training of LLMs through Truncated SVD Layers
di: Kamali, Kaivan, et al.
Pubblicazione: (2026)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
di: Belenki, Lior, et al.
Pubblicazione: (2025)

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models
di: Tang, Anke, et al.
Pubblicazione: (2024)

LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
di: Chang, Ching, et al.
Pubblicazione: (2023)

Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
di: Zhu, Shuchen, et al.
Pubblicazione: (2026)

MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
di: Kang, Jikun, et al.
Pubblicazione: (2024)

PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
di: Zhang, Ze Yu, et al.
Pubblicazione: (2025)

BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
di: Jin, Zewen, et al.
Pubblicazione: (2025)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
di: Panda, Ashwinee, et al.
Pubblicazione: (2025)

MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators
di: Huang, Beichen, et al.
Pubblicazione: (2025)

MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning
di: Zhang, Yutong, et al.
Pubblicazione: (2026)

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
di: Wang, Hong, et al.
Pubblicazione: (2025)

MoTE: Mixture of Task-specific Experts for Pre-Trained ModelBased Class-incremental Learning
di: Li, Linjie, et al.
Pubblicazione: (2025)

Heterogeneous Low-Bandwidth Pre-Training of LLMs
di: Obeidi, Yazan, et al.
Pubblicazione: (2026)

Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services
di: Wang, Weiyan, et al.
Pubblicazione: (2024)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
di: Jiang, Yukun, et al.
Pubblicazione: (2026)

Exploiting Adaptive Channel Pruning for Communication-Efficient Split Learning
di: Tan, Jialei, et al.
Pubblicazione: (2026)

CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation
di: Chen, Yutong, et al.
Pubblicazione: (2026)

CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
di: Wang, Xinze, et al.
Pubblicazione: (2025)

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
di: Pan, Xinglin, et al.
Pubblicazione: (2025)

Efficient Mixture Learning in Black-Box Variational Inference
di: Hotti, Alexandra, et al.
Pubblicazione: (2024)

CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
di: Liu, Ziyue, et al.
Pubblicazione: (2025)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
di: Cai, Weilin, et al.
Pubblicazione: (2024)

Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs
di: Luo, Beier, et al.
Pubblicazione: (2026)

Efficient Sparse Training with Structured Dropout
di: Lo, Andy
Pubblicazione: (2024)

MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
di: Zhou, Ruijie, et al.
Pubblicazione: (2026)