Apriel-H1: Towards Efficient Enterprise Reasoning Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909887106121728 |
|---|---|
| author | Ostapenko, Oleksiy Kumar, Luke Li, Raymond Kocetkov, Denis Lamy-Poirier, Joel Radhakrishna, Shruthan Parikh, Soham Mishra, Shambhavi Paquet, Sebastien Sunkara, Srinivas Bécaert, Valérie Madhusudhan, Sathwik Tejaswi Scholak, Torsten |
| author_facet | Ostapenko, Oleksiy Kumar, Luke Li, Raymond Kocetkov, Denis Lamy-Poirier, Joel Radhakrishna, Shruthan Parikh, Soham Mishra, Shambhavi Paquet, Sebastien Sunkara, Srinivas Bécaert, Valérie Madhusudhan, Sathwik Tejaswi Scholak, Torsten |
| contents | Large Language Models (LLMs) achieve remarkable reasoning capabilities through transformer architectures with attention mechanisms. However, transformers suffer from quadratic time and memory complexity in the attention module (MHA) and require caching key-value states during inference, which severely limits throughput and scalability. High inference throughput is critical for agentic tasks, long-context reasoning, efficient deployment under high request loads, and more efficient test-time compute scaling.
State Space Models (SSMs) such as Mamba offer a promising alternative with linear inference complexity and a constant memory footprint via recurrent computation with fixed-size hidden states. In this technical report we introduce the Apriel-H1 family of hybrid LLMs that combine transformer attention and SSM sequence mixers for efficient reasoning at 15B model size. These models are obtained through incremental distillation from a pretrained reasoning transformer, Apriel-Nemotron-15B-Thinker, progressively replacing less critical attention layers with linear Mamba blocks.
We release multiple post-distillation variants of Apriel-H1-15B-Thinker with different SSM-to-MHA ratios and analyse how reasoning performance degrades as more Mamba layers replace MHA. Additionally, we release a 30/50 hybrid variant of Apriel-H1, further fine-tuned on a supervised dataset of reasoning traces, achieving over 2x higher inference throughput when deployed in the production-ready vLLM environment, with minimal degradation in reasoning performance. This shows that distilled hybrid SSM-Transformer architectures can deliver substantial efficiency gains over the pretrained transformer equivalent without substantially compromising the reasoning quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02651 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Apriel-H1: Towards Efficient Enterprise Reasoning Models Ostapenko, Oleksiy Kumar, Luke Li, Raymond Kocetkov, Denis Lamy-Poirier, Joel Radhakrishna, Shruthan Parikh, Soham Mishra, Shambhavi Paquet, Sebastien Sunkara, Srinivas Bécaert, Valérie Madhusudhan, Sathwik Tejaswi Scholak, Torsten Machine Learning Artificial Intelligence Large Language Models (LLMs) achieve remarkable reasoning capabilities through transformer architectures with attention mechanisms. However, transformers suffer from quadratic time and memory complexity in the attention module (MHA) and require caching key-value states during inference, which severely limits throughput and scalability. High inference throughput is critical for agentic tasks, long-context reasoning, efficient deployment under high request loads, and more efficient test-time compute scaling. State Space Models (SSMs) such as Mamba offer a promising alternative with linear inference complexity and a constant memory footprint via recurrent computation with fixed-size hidden states. In this technical report we introduce the Apriel-H1 family of hybrid LLMs that combine transformer attention and SSM sequence mixers for efficient reasoning at 15B model size. These models are obtained through incremental distillation from a pretrained reasoning transformer, Apriel-Nemotron-15B-Thinker, progressively replacing less critical attention layers with linear Mamba blocks. We release multiple post-distillation variants of Apriel-H1-15B-Thinker with different SSM-to-MHA ratios and analyse how reasoning performance degrades as more Mamba layers replace MHA. Additionally, we release a 30/50 hybrid variant of Apriel-H1, further fine-tuned on a supervised dataset of reasoning traces, achieving over 2x higher inference throughput when deployed in the production-ready vLLM environment, with minimal degradation in reasoning performance. This shows that distilled hybrid SSM-Transformer architectures can deliver substantial efficiency gains over the pretrained transformer equivalent without substantially compromising the reasoning quality. |
| title | Apriel-H1: Towards Efficient Enterprise Reasoning Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.02651 |