Cover Image

Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mehta, Sushant, Dandekar, Raj, Dandekar, Rajat, Panat, Sreedath
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Latent Multi-Head Attention for Small Language Models
by: Mehta, Sushant, et al.
Published: (2025)

Muon: Training and Trade-offs with Latent Attention and MoE
by: Mehta, Sushant, et al.
Published: (2025)

Vision-Language Models display a strong gender bias
by: Konavoor, Aiswarya, et al.
Published: (2025)

Decoders Laugh as Loud as Encoders
by: Borodach, Eli, et al.
Published: (2025)

Modeling chaotic Lorenz ODE System using Scientific Machine Learning
by: Kashyap, Sameera S, et al.
Published: (2024)

Scientific machine learning in ecological systems: A study on the predator-prey dynamics
by: Devgupta, Ranabir, et al.
Published: (2024)

NanoVLMs: How small can we go and still make coherent Vision Language Models?
by: Agarwalla, Mukund, et al.
Published: (2025)

Beyond Passive Viewing: A Pilot Study of a Hybrid Learning Platform Augmenting Video Lectures with Conversational AI
by: Abraar, Mohammed, et al.
Published: (2026)

CBEval: A framework for evaluating and interpreting cognitive biases in LLMs
by: Shaikh, Ammar, et al.
Published: (2024)

A comparative study of NeuralODE and Universal ODE approaches to solving Chandrasekhar White Dwarf equation
by: Martinez, Raymundo Vazquez, et al.
Published: (2024)

A Scientific Machine Learning Approach for Predicting and Forecasting Battery Degradation in Electric Vehicles
by: Murgai, Sharv, et al.
Published: (2024)

Simulating Misinformation Propagation in Social Networks using Large Language Models
by: Maurya, Raj Gaurav, et al.
Published: (2025)

Forecasting N-Body Dynamics: A Comparative Study of Neural Ordinary Differential Equations and Universal Differential Equations
by: S, Suriya R, et al.
Published: (2025)

HULLMI: Human vs LLM identification with explainability
by: Joshi, Prathamesh Dinesh, et al.
Published: (2024)

EARS-UDE: Evaluating Auditory Response in Sensory Overload with Universal Differential Equations
by: Salunke, Miheer, et al.
Published: (2025)

Adaptive tumor growth forecasting via neural & universal ODEs
by: Subramanian, Kavya, et al.
Published: (2025)

BULL-ODE: Bullwhip Learning with Neural ODEs and Universal Differential Equations under Stochastic Demand
by: Naik, Nachiket N., et al.
Published: (2025)

Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English
by: Dawson, Fiifi, et al.
Published: (2024)

A study of Universal ODE approaches to predicting soil organic carbon
by: V. V, Satyanarayana Raju G., et al.
Published: (2025)

Three methods, one problem: Classical and AI approaches to no-three-in-line
by: Ramanathan, Pranav, et al.
Published: (2025)

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
by: Patil, Nirvan, et al.
Published: (2025)

Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics
by: Kainth, Kamalpreet Singh, et al.
Published: (2025)

Understanding Malware Propagation Dynamics through Scientific Machine Learning
by: Pappu, Karthik, et al.
Published: (2025)

Physical Informed Neural Networks for modeling ocean pollutant
by: Battina, Karishma, et al.
Published: (2025)

Model-Grounded Symbolic Artificial Intelligence Systems Learning and Reasoning with Model-Grounded Symbolic Artificial Intelligence Systems
by: Chattopadhyay, Aniruddha, et al.
Published: (2025)

When Are Learning Biases Equivalent? A Unifying Framework for Fairness, Robustness, and Distribution Shift
by: Mehta, Sushant
Published: (2025)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
by: Li, Cheng, et al.
Published: (2025)

Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems
by: Mehta, Sushant
Published: (2025)

MATRAG: Multi-Agent Transparent Retrieval-Augmented Generation for Explainable Recommendations
by: Mehta, Sushant
Published: (2026)

Multi-Head Mixture-of-Experts
by: Wu, Xun, et al.
Published: (2024)

Scaling Laws and In-Context Learning: A Unified Theoretical Framework
by: Mehta, Sushant, et al.
Published: (2025)

FASST: Fast LLM-based Simultaneous Speech Translation
by: Ouyang, Siqi, et al.
Published: (2024)

Beyond Surface-Level Similarity: Hierarchical Contamination Detection for Synthetic Training Data in Foundation Models
by: Mehta, Sushant
Published: (2025)

HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems
by: Mehta, Sushant
Published: (2026)

Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
by: Zhu, Shien, et al.
Published: (2025)

MoH: Multi-Head Attention as Mixture-of-Head Attention
by: Jin, Peng, et al.
Published: (2024)

Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
by: Yun, Sungmin, et al.
Published: (2025)

A Descriptive and Normative Theory of Human Beliefs in RLHF
by: Dandekar, Sylee, et al.
Published: (2025)

Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers
by: Xu, Boxun, et al.
Published: (2024)

Efficiently Editing Mixture-of-Experts Models with Compressed Experts
by: He, Yifei, et al.
Published: (2025)