Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909915674574848 |
|---|---|
| author | Taghibakhshi, Ali Sreenivas, Sharath Turuvekere Muralidharan, Saurav Cai, Ruisi Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Suhara, Yoshi Olabiyi, Oluwatobi Korzekwa, Daniel Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan Aithal, Ashwath Tajbakhsh, Nima Molchanov, Pavlo |
| author_facet | Taghibakhshi, Ali Sreenivas, Sharath Turuvekere Muralidharan, Saurav Cai, Ruisi Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Suhara, Yoshi Olabiyi, Oluwatobi Korzekwa, Daniel Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan Aithal, Ashwath Tajbakhsh, Nima Molchanov, Pavlo |
| contents | Training a family of large language models targeting multiple scales and deployment objectives is prohibitively expensive, requiring separate training runs for each different size. Recent work on model compression through pruning and knowledge distillation has reduced this cost; however, this process still incurs hundreds of billions of tokens worth of training cost per compressed model. In this paper, we present Nemotron Elastic, a framework for building reasoning-oriented LLMs, including hybrid Mamba-Attention architectures, that embed multiple nested submodels within a single parent model, each optimized for different deployment configurations and budgets. Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment without additional training or fine-tuning. We enable this functionality through an end-to-end trained router, tightly coupled to a two-stage training curriculum designed specifically for reasoning models. We additionally introduce group-aware SSM elastification that preserves Mamba's structural constraints, heterogeneous MLP elastification, normalized MSE-based layer importance for improved depth selection, and knowledge distillation enabling simultaneous multi-budget optimization. We apply Nemotron Elastic to the Nemotron Nano V2 12B model, simultaneously producing a 9B and a 6B model using only 110B training tokens; this results in over 360x cost reduction compared to training model families from scratch, and around 7x compared to SoTA compression techniques. Each of the nested models performs on par or better than the SoTA in accuracy. Moreover, unlike other compression methods, the nested capability of our approach allows having a many-in-one reasoning model that has constant deployment memory against the number of models in the family. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16664 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs Taghibakhshi, Ali Sreenivas, Sharath Turuvekere Muralidharan, Saurav Cai, Ruisi Chochowski, Marcin Mahabaleshwarkar, Ameya Sunil Suhara, Yoshi Olabiyi, Oluwatobi Korzekwa, Daniel Patwary, Mostofa Shoeybi, Mohammad Kautz, Jan Catanzaro, Bryan Aithal, Ashwath Tajbakhsh, Nima Molchanov, Pavlo Computation and Language Training a family of large language models targeting multiple scales and deployment objectives is prohibitively expensive, requiring separate training runs for each different size. Recent work on model compression through pruning and knowledge distillation has reduced this cost; however, this process still incurs hundreds of billions of tokens worth of training cost per compressed model. In this paper, we present Nemotron Elastic, a framework for building reasoning-oriented LLMs, including hybrid Mamba-Attention architectures, that embed multiple nested submodels within a single parent model, each optimized for different deployment configurations and budgets. Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment without additional training or fine-tuning. We enable this functionality through an end-to-end trained router, tightly coupled to a two-stage training curriculum designed specifically for reasoning models. We additionally introduce group-aware SSM elastification that preserves Mamba's structural constraints, heterogeneous MLP elastification, normalized MSE-based layer importance for improved depth selection, and knowledge distillation enabling simultaneous multi-budget optimization. We apply Nemotron Elastic to the Nemotron Nano V2 12B model, simultaneously producing a 9B and a 6B model using only 110B training tokens; this results in over 360x cost reduction compared to training model families from scratch, and around 7x compared to SoTA compression techniques. Each of the nested models performs on par or better than the SoTA in accuracy. Moreover, unlike other compression methods, the nested capability of our approach allows having a many-in-one reasoning model that has constant deployment memory against the number of models in the family. |
| title | Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.16664 |