Cover Image

Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Experts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Xin, Sarwate, Anand
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling
by: Li, Xin, et al.
Published: (2024)

LSR-Adapt: Ultra-Efficient Parameter Tuning with Matrix Low Separation Rank Kernel Adaptation
by: Li, Xin, et al.
Published: (2025)

Learning To Help: Training Models to Assist Legacy Devices
by: Wu, Yu, et al.
Published: (2024)

Understanding Generative AI Content with Embedding Models
by: Vargas, Max, et al.
Published: (2024)

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
by: Chen, Shengzhuang, et al.
Published: (2025)

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
by: Li, Ziyue, et al.
Published: (2024)

Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
by: Kawata, Ryotaro, et al.
Published: (2025)

Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
by: Li, Zhongyang, et al.
Published: (2025)

Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
by: Nguyen-Nhat, Minh-Khoi, et al.
Published: (2025)

Expert Merging in Sparse Mixture of Experts with Nash Bargaining
by: Nguyen, Dung V., et al.
Published: (2025)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
by: Ahrac, Sagi, et al.
Published: (2026)

Learning to Help in Multi-Class Settings
by: Wu, Yu, et al.
Published: (2025)

On the Role of Discrete Representation in Sparse Mixture of Experts
by: Do, Giang, et al.
Published: (2024)

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
by: Shihab, Ibne Farabi, et al.
Published: (2026)

Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning
by: Siddika, Fatema, et al.
Published: (2026)

Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
by: Nguyen, Duc Anh, et al.
Published: (2025)

Improving Routing in Sparse Mixture of Experts with Graph of Tokens
by: Nguyen, Tam, et al.
Published: (2025)

Mixture of Latent Experts Using Tensor Products
by: Su, Zhan, et al.
Published: (2024)

Discretized Gradient Flow for Manifold Learning in the Space of Embeddings
by: Gold, Dara, et al.
Published: (2019)

Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
by: Nikolic, Strahinja, et al.
Published: (2025)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
by: Zhang, Zeliang, et al.
Published: (2024)

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
by: Chowdhury, Mohammed Nowaz Rabbani, et al.
Published: (2024)

One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning
by: Le, Minh, et al.
Published: (2025)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
by: Jiang, Yukun, et al.
Published: (2026)

Measuring training variability from stochastic optimization using robust nonparametric testing
by: Banerjee, Sinjini, et al.
Published: (2024)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
by: Panda, Ashwinee, et al.
Published: (2025)

CoSMoEs: Compact Sparse Mixture of Experts
by: Huber, Patrick, et al.
Published: (2025)

Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings
by: Hallee, Logan, et al.
Published: (2024)

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
by: Muzio, Alexandre, et al.
Published: (2024)

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts
by: Park, Sejik
Published: (2024)

Adversarial Purification by Consistency-aware Latent Space Optimization on Data Manifolds
by: Zhang, Shuhai, et al.
Published: (2024)

CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
by: Pham, Quang, et al.
Published: (2024)

From Sparse to Soft Mixtures of Experts
by: Puigcerver, Joan, et al.
Published: (2023)

Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2023)

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
by: Zhao, Jinze, et al.
Published: (2024)

MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
by: Zeng, Runjia, et al.
Published: (2025)

Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models
by: Kachuee, Sajjad, et al.
Published: (2026)

Soft-to-Hard Routing in Sparse Mixture-of-Experts Models
by: Rastegar, Reza
Published: (2026)

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
by: Liu, Enshu, et al.
Published: (2024)

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
by: Pan, Xinglin, et al.
Published: (2025)