Cover Image

Hymba: A Hybrid-head Architecture for Small Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dong, Xin, Fu, Yonggan, Diao, Shizhe, Byeon, Wonmin, Chen, Zijia, Mahabaleshwarkar, Ameya Sunil, Liu, Shih-Yang, Van Keirsbilck, Matthijs, Chen, Min-Hung, Suhara, Yoshi, Lin, Yingyan, Kautz, Jan, Molchanov, Pavlo
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

When2Call: When (not) to Call Tools
by: Ross, Hayley, et al.
Published: (2025)

Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
by: Fu, Yonggan, et al.
Published: (2025)

Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
by: Diao, Shizhe, et al.
Published: (2025)

Small Language Models are the Future of Agentic AI
by: Belcak, Peter, et al.
Published: (2025)

LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
by: Ye, Zhifan, et al.
Published: (2025)

LLM Pruning and Distillation in Practice: The Minitron Approach
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2024)

Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
by: Taghibakhshi, Ali, et al.
Published: (2025)

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
by: Taghibakhshi, Ali, et al.
Published: (2025)

Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
by: Fu, Yonggan, et al.
Published: (2025)

Source Identification in Abstractive Summarization
by: Suhara, Yoshi, et al.
Published: (2024)

Stateful Token Reduction for Long-Video Hybrid VLMs
by: Jiang, Jindong, et al.
Published: (2026)

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
by: Taghibakhshi, Ali, et al.
Published: (2026)

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
by: Shi, Dachuan, et al.
Published: (2025)

Fast-dLLM v2: Efficient Block-Diffusion LLM
by: Wu, Chengyue, et al.
Published: (2025)

Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation
by: Belcak, Peter, et al.
Published: (2025)

AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One
by: Ranzinger, Mike, et al.
Published: (2023)

DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning
by: Liu, Shih-Yang, et al.
Published: (2025)

Noisy Pairing and Partial Supervision for Stylized Opinion Summarization
by: Iso, Hayate, et al.
Published: (2022)

Large Language Models are Inconsistent and Biased Evaluators
by: Stureborg, Rickard, et al.
Published: (2024)

ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
by: Wang, Zhilin, et al.
Published: (2025)

$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
by: Du, Zhenbang, et al.
Published: (2026)

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
by: Liu, Shih-Yang, et al.
Published: (2026)

BroRL: Scaling Reinforcement Learning via Broadened Exploration
by: Hu, Jian, et al.
Published: (2025)

TiDAR: Think in Diffusion, Talk in Autoregression
by: Liu, Jingyu, et al.
Published: (2025)

Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
by: Fu, Yonggan, et al.
Published: (2020)

ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
by: Su, Hongjin, et al.
Published: (2025)

Universal Deep Research: Bring Your Own Model and Strategy
by: Belcak, Peter, et al.
Published: (2025)

GSPN-2: Efficient Parallel Sequence Modeling
by: Wang, Hongjun, et al.
Published: (2025)

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning
by: Kang, Minki, et al.
Published: (2026)

Entropy-Regularized Process Reward Model
by: Zhang, Hanning, et al.
Published: (2024)

ComHymba: Low-Complexity Domain-Informed Foundation Model for Wireless Communications
by: Yang, Bowen, et al.
Published: (2026)

Your Large Language Models Are Leaving Fingerprints
by: McGovern, Hope, et al.
Published: (2024)

FeatSharp: Your Vision Model Features, Sharper
by: Ranzinger, Mike, et al.
Published: (2025)

TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces
by: Quétant, Guillaume, et al.
Published: (2025)

Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
by: Jiang, Yitong, et al.
Published: (2026)

LITA: Language Instructed Temporal-Localization Assistant
by: Huang, De-An, et al.
Published: (2024)

Flextron: Many-in-One Flexible Large Language Model
by: Cai, Ruisi, et al.
Published: (2024)

Parallel Sequence Modeling via Generalized Spatial Propagation Network
by: Wang, Hongjun, et al.
Published: (2025)

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
by: Park, Dongmin, et al.
Published: (2025)

DoRA: Weight-Decomposed Low-Rank Adaptation
by: Liu, Shih-Yang, et al.
Published: (2024)