Copertina

UMoE: Unifying Attention and FFN with Shared Experts

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yang, Yuanhang, Wang, Chaozheng, Li, Jing
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks
di: Liu, Ningyuan, et al.
Pubblicazione: (2025)

Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads
di: Song, Chendong, et al.
Pubblicazione: (2026)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
di: Li, Cheng, et al.
Pubblicazione: (2025)

Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
di: Pei, Zehua, et al.
Pubblicazione: (2025)

Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
di: Smithline, Gabriel, et al.
Pubblicazione: (2026)

Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity
di: Gautam, Aayush, et al.
Pubblicazione: (2026)

Translating Expert Intuition into Quantifiable Features: Encode Investigator Domain Knowledge via LLM for Enhanced Predictive Analytics
di: Jing, Phoebe, et al.
Pubblicazione: (2024)

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
di: Jaiswal, Ajay, et al.
Pubblicazione: (2025)

Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting
di: Zhao, Yanjun, et al.
Pubblicazione: (2024)

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
di: Huang, Minbin, et al.
Pubblicazione: (2026)

How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
di: Wu, Hanjiang, et al.
Pubblicazione: (2026)

A Shared Low-Rank Adaptation Approach to Personalized RLHF
di: Liu, Renpu, et al.
Pubblicazione: (2025)

Quaternion Self-Attention with Shared Scores
di: Yamauchi, Shogo, et al.
Pubblicazione: (2026)

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
di: Li, Junzhuo, et al.
Pubblicazione: (2026)

BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
di: Wang, Yun, et al.
Pubblicazione: (2025)

LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
di: Li, Wenbing, et al.
Pubblicazione: (2025)

XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
di: Yang, Yuanhang, et al.
Pubblicazione: (2024)

UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts
di: Wang, Fu-Yun, et al.
Pubblicazione: (2025)

IDInit: A Universal and Stable Initialization Method for Neural Network Training
di: Pan, Yu, et al.
Pubblicazione: (2025)

Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
di: Yang, Minghao, et al.
Pubblicazione: (2025)

Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction
di: Yuan, Li, et al.
Pubblicazione: (2025)

Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
di: Athiwaratkun, Ben, et al.
Pubblicazione: (2024)

MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
di: Jin, Peng, et al.
Pubblicazione: (2024)

MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
di: Chou, Yuhong, et al.
Pubblicazione: (2024)

TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
di: Kunwar, Pradip, et al.
Pubblicazione: (2025)

Attention Needs to Focus: A Unified Perspective on Attention Allocation
di: Fu, Zichuan, et al.
Pubblicazione: (2026)

MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Prediction
di: Wang, Xiaoyang, et al.
Pubblicazione: (2025)

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
di: Yang, Cheng, et al.
Pubblicazione: (2024)

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
di: Zhao, Ziyu, et al.
Pubblicazione: (2025)

Unified Class and Domain Incremental Learning with Mixture of Experts for Indoor Localization
di: Singampalli, Akhil, et al.
Pubblicazione: (2025)

CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting
di: Pati, Viresh, et al.
Pubblicazione: (2026)

XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference
di: Vankov, Daniil, et al.
Pubblicazione: (2026)

FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression
di: Gao, Yifei, et al.
Pubblicazione: (2025)

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
di: Hao, Jiawei, et al.
Pubblicazione: (2026)

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
di: Zhao, Hao, et al.
Pubblicazione: (2024)

Retro-Expert: Collaborative Reasoning for Interpretable Retrosynthesis
di: Li, Xinyi, et al.
Pubblicazione: (2025)

EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
di: Chen, Yuanteng, et al.
Pubblicazione: (2025)

SD-MoE: Spectral Decomposition for Effective Expert Specialization
di: Huang, Ruijun, et al.
Pubblicazione: (2026)

Taxon: Hierarchical Tax Code Prediction with Semantically Aligned LLM Expert Guidance
di: Li, Jihang, et al.
Pubblicazione: (2026)

Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
di: Yan, Fanqi, et al.
Pubblicazione: (2025)