Cover Image

Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Cai, Ruisi, Ro, Yeonju, Kim, Geon-Woo, Wang, Peihao, Bejnordi, Babak Ehteshami, Akella, Aditya, Wang, Zhangyang
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
by: Ro, Yeonju, et al.
Published: (2025)

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
by: Zhao, Jinze, et al.
Published: (2024)

Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration
by: Shen, Jucheng, et al.
Published: (2025)

Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems
by: Zhu, Jianing, et al.
Published: (2026)

Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
by: Skliar, Andrii, et al.
Published: (2024)

Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
by: Zhao, Jinze, et al.
Published: (2024)

Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
by: Mirvakhabova, Leyla, et al.
Published: (2025)

KaVa: Latent Reasoning via Compressed KV-Cache Distillation
by: Kuzina, Anna, et al.
Published: (2025)

Sherlock: Reliable and Efficient Agentic Workflow Execution
by: Ro, Yeonju, et al.
Published: (2025)

LoCoCo: Dropping In Convolutions for Long Context Compression
by: Cai, Ruisi, et al.
Published: (2024)

HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
by: Kim, Geon-Woo, et al.
Published: (2025)

When Do Graph Foundation Models Transfer? A Data-Centric Theory
by: Zhu, Jiajun, et al.
Published: (2026)

Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
by: Ran, Junfeng, et al.
Published: (2025)

Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding
by: Zhu, Jiajun, et al.
Published: (2025)

Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony
by: Wang, Shaoyu, et al.
Published: (2025)

Layerwise Recurrent Router for Mixture-of-Experts
by: Qiu, Zihan, et al.
Published: (2024)

Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning
by: Wang, Peihao, et al.
Published: (2025)

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
by: Jaiswal, Ajay, et al.
Published: (2024)

Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
by: Wang, Peihao, et al.
Published: (2024)

$\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space
by: Wang, Peihao, et al.
Published: (2026)

Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
by: Bergner, Benjamin, et al.
Published: (2024)

InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning
by: Bejnordi, Babak Ehteshami, et al.
Published: (2024)

Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models
by: Shen, Jucheng, et al.
Published: (2025)

Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier
by: Rha, Hyeongseop, et al.
Published: (2025)

Position: Weight Space Should Be a First-Class Generative AI Modality
by: Wang, Zhangyang, et al.
Published: (2026)

Routers in Vision Mixture of Experts: An Empirical Study
by: Liu, Tianlin, et al.
Published: (2024)

RouterKT: Mixture-of-Experts for Knowledge Tracing
by: Liao, Han, et al.
Published: (2025)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
by: Lv, Ang, et al.
Published: (2025)

FlexGS: Train Once, Deploy Everywhere with Many-in-One Flexible 3D Gaussian Splatting
by: Liu, Hengyu, et al.
Published: (2025)

Yuan 2.0-M32: Mixture of Experts with Attention Router
by: Wu, Shaohua, et al.
Published: (2024)

A Self-Supervised Mixture-of-Experts Framework for Multi-behavior Recommendation
by: Kim, Kyungho, et al.
Published: (2025)

Load Balancing Mixture of Experts with Similarity Preserving Routers
by: Omi, Nabil, et al.
Published: (2025)

Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2024)

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models
by: Antoine, Elie, et al.
Published: (2024)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
by: Ahrac, Sagi, et al.
Published: (2026)

Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
by: Chen, Hao Mark, et al.
Published: (2026)

CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
by: Kim, Junho, et al.
Published: (2024)

OMEGA: A Low-Latency GNN Serving System for Large Graphs
by: Kim, Geon-Woo, et al.
Published: (2025)

On the Fundamental Limitations of Decentralized Learnable Reward Shaping in Cooperative Multi-Agent Reinforcement Learning
by: Akella, Aditya
Published: (2025)

Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
by: Zheng, Chuanyang, et al.
Published: (2025)