Cover Image

Muon: Training and Trade-offs with Latent Attention and MoE

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mehta, Sushant, Dandekar, Raj, Dandekar, Rajat, Panat, Sreedath
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Latent Multi-Head Attention for Small Language Models
by: Mehta, Sushant, et al.
Published: (2025)

Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
by: Mehta, Sushant, et al.
Published: (2025)

Scientific machine learning in ecological systems: A study on the predator-prey dynamics
by: Devgupta, Ranabir, et al.
Published: (2024)

Modeling chaotic Lorenz ODE System using Scientific Machine Learning
by: Kashyap, Sameera S, et al.
Published: (2024)

A comparative study of NeuralODE and Universal ODE approaches to solving Chandrasekhar White Dwarf equation
by: Martinez, Raymundo Vazquez, et al.
Published: (2024)

A Scientific Machine Learning Approach for Predicting and Forecasting Battery Degradation in Electric Vehicles
by: Murgai, Sharv, et al.
Published: (2024)

Forecasting N-Body Dynamics: A Comparative Study of Neural Ordinary Differential Equations and Universal Differential Equations
by: S, Suriya R, et al.
Published: (2025)

Physical Informed Neural Networks for modeling ocean pollutant
by: Battina, Karishma, et al.
Published: (2025)

Understanding Malware Propagation Dynamics through Scientific Machine Learning
by: Pappu, Karthik, et al.
Published: (2025)

Adaptive tumor growth forecasting via neural & universal ODEs
by: Subramanian, Kavya, et al.
Published: (2025)

BULL-ODE: Bullwhip Learning with Neural ODEs and Universal Differential Equations under Stochastic Demand
by: Naik, Nachiket N., et al.
Published: (2025)

Decoders Laugh as Loud as Encoders
by: Borodach, Eli, et al.
Published: (2025)

A study of Universal ODE approaches to predicting soil organic carbon
by: V. V, Satyanarayana Raju G., et al.
Published: (2025)

Vision-Language Models display a strong gender bias
by: Konavoor, Aiswarya, et al.
Published: (2025)

CBEval: A framework for evaluating and interpreting cognitive biases in LLMs
by: Shaikh, Ammar, et al.
Published: (2024)

Beyond Passive Viewing: A Pilot Study of a Hybrid Learning Platform Augmenting Video Lectures with Conversational AI
by: Abraar, Mohammed, et al.
Published: (2026)

NanoVLMs: How small can we go and still make coherent Vision Language Models?
by: Agarwalla, Mukund, et al.
Published: (2025)

Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics
by: Kainth, Kamalpreet Singh, et al.
Published: (2025)

Simulating Misinformation Propagation in Social Networks using Large Language Models
by: Maurya, Raj Gaurav, et al.
Published: (2025)

EARS-UDE: Evaluating Auditory Response in Sensory Overload with Universal Differential Equations
by: Salunke, Miheer, et al.
Published: (2025)

HULLMI: Human vs LLM identification with explainability
by: Joshi, Prathamesh Dinesh, et al.
Published: (2024)

Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English
by: Dawson, Fiifi, et al.
Published: (2024)

Three methods, one problem: Classical and AI approaches to no-three-in-line
by: Ramanathan, Pranav, et al.
Published: (2025)

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
by: Patil, Nirvan, et al.
Published: (2025)

Beyond Surface-Level Similarity: Hierarchical Contamination Detection for Synthetic Training Data in Foundation Models
by: Mehta, Sushant
Published: (2025)

Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
by: Cui, Chenwei, et al.
Published: (2026)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
by: Fu, Zizhuo, et al.
Published: (2026)

EMO: Frustratingly Easy Progressive Training of Extendable MoE
by: Jin, Linghao, et al.
Published: (2026)

When Are Learning Biases Equivalent? A Unifying Framework for Fairness, Robustness, and Distribution Shift
by: Mehta, Sushant
Published: (2025)

Mol-MoE: Training Preference-Guided Routers for Molecule Generation
by: Calanzone, Diego, et al.
Published: (2025)

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
by: Chen, Anrui, et al.
Published: (2026)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
by: Hannah, Lauren. A, et al.
Published: (2025)

STAMImputer: Spatio-Temporal Attention MoE for Traffic Data Imputation
by: Wang, Yiming, et al.
Published: (2025)

GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
by: Wu, Haoze, et al.
Published: (2024)

A Descriptive and Normative Theory of Human Beliefs in RLHF
by: Dandekar, Sylee, et al.
Published: (2025)

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
by: Liu, Dennis, et al.
Published: (2025)

MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
by: Xue, Leyang, et al.
Published: (2024)

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
by: Manzoni, Andrea
Published: (2026)

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
by: Du, Xianzhi, et al.
Published: (2024)

Grouter: Decoupling Routing from Representation for Accelerated MoE Training
by: Xu, Yuqi, et al.
Published: (2026)