Cover Image

HMoE: Heterogeneous Mixture of Experts for Language Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, An, Sun, Xingwu, Xie, Ruobing, Li, Shuaipeng, Zhu, Jiaqi, Yang, Zhen, Zhao, Pinxue, Han, J. N., Kang, Zhanhui, Wang, Di, Okazaki, Naoaki, Xu, Cheng-zhong
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Towards a Comprehensive Scaling Law of Mixture-of-Experts
by: Zhao, Guoliang, et al.
Published: (2025)

TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
by: Li, Yixing, et al.
Published: (2025)

More Expressive Attention with Negative Weights
by: Lv, Ang, et al.
Published: (2024)

Exploring Forgetting in Large Language Model Pre-Training
by: Liao, Chonghua, et al.
Published: (2024)

Autonomy-of-Experts Models
by: Lv, Ang, et al.
Published: (2025)

Language Models "Grok" to Copy
by: Lv, Ang, et al.
Published: (2024)

The Elephant in the Room: Rethinking the Usage of Pre-trained Language Model in Sequential Recommendation
by: Qu, Zekai, et al.
Published: (2024)

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
by: Fu, Yuhan, et al.
Published: (2024)

Continuous Speech Tokenizer in Text To Speech
by: Li, Yixing, et al.
Published: (2024)

Self-Distillation for Multi-Token Prediction
by: Zhao, Guoliang, et al.
Published: (2026)

The Security Threat of Compressed Projectors in Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2025)

QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2025)

Lossless KV Cache Compression to 2%
by: Yang, Zhen, et al.
Published: (2024)

The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
by: Lv, Ang, et al.
Published: (2025)

DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
by: Zhang, Yudong, et al.
Published: (2024)

Enhancing Contrastive Learning Inspired by the Philosophy of "The Blind Men and the Elephant"
by: Zhang, Yudong, et al.
Published: (2024)

Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs
by: Zhang, Yudong, et al.
Published: (2025)

Multi-Grained Patch Training for Efficient LLM-based Recommendation
by: Liao, Jiayi, et al.
Published: (2025)

Hybrid-Tower: Fine-grained Pseudo-query Interaction and Generation for Text-to-Video Retrieval
by: Lan, Bangxiang, et al.
Published: (2025)

Scaling Laws for Floating Point Quantization Training
by: Sun, Xingwu, et al.
Published: (2025)

Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
by: Fu, Yuhan, et al.
Published: (2024)

RosePO: Aligning LLM-based Recommenders with Human Values
by: Liao, Jiayi, et al.
Published: (2024)

Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training
by: Ma, Haokai, et al.
Published: (2025)

PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset
by: Liu, Jiazhen, et al.
Published: (2024)

Proximal Supervised Fine-Tuning
by: Zhu, Wenhong, et al.
Published: (2025)

Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning
by: Chen, Zhongzhi, et al.
Published: (2023)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
by: Song, Guanghui, et al.
Published: (2025)

Negative Sampling in Recommendation: A Survey and Future Directions
by: Ma, Haokai, et al.
Published: (2024)

PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions
by: Zhang, Yudong, et al.
Published: (2024)

Knowledge of Pretrained Language Models on Surface Information of Tokens
by: Hiraoka, Tatsuya, et al.
Published: (2024)

From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
by: Ma, Youmi, et al.
Published: (2026)

JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
by: Maeda, Koki, et al.
Published: (2026)

Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer
by: Ma, Youmi, et al.
Published: (2024)

Mixture of Heterogeneous Grouped Experts for Language Modeling
by: Ma, Zhicheng, et al.
Published: (2026)

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels
by: Anantaprayoon, Panatchakorn, et al.
Published: (2023)

Tokenization as Finite-State Transduction
by: Cognetta, Marco, et al.
Published: (2024)

Drifting Objectives for Refining Discrete Diffusion Language Models
by: Oba, Daisuke, et al.
Published: (2026)

Diffusion-State Policy Optimization for Masked Diffusion Language Models
by: Oba, Daisuke, et al.
Published: (2026)

Social Bias Evaluation for Large Language Models Requires Prompt Variations
by: Hida, Rem, et al.
Published: (2024)

Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Models
by: Anantaprayoon, Panatchakorn, et al.
Published: (2025)