Staff View: :: Library Catalog

Saved in:

Bibliographic Details
Main Authors:	NVIDIA, :, Blakeman, Aaron, Basant, Aarti, Khattar, Abhinav, Renduchintala, Adithya, Bercovich, Akhiad, Ficek, Aleksander, Bjorlin, Alexis, Taghibakhshi, Ali, Deshmukh, Amala Sanjay, Mahabaleshwarkar, Ameya Sunil, Tao, Andrew, Shors, Anna, Aithal, Ashwath, Poojary, Ashwin, Dattagupta, Ayush, Buddharaju, Balaram, Chen, Bobby, Ginsburg, Boris, Wang, Boxin, Norick, Brandon, Butterfield, Brian, Catanzaro, Bryan, del Mundo, Carlo, Dong, Chengyu, Harvey, Christine, Parisien, Christopher, Su, Dan, Korzekwa, Daniel, Yin, Danny, Gitman, Daria, Mosallanezhad, David, Narayanan, Deepak, Fridman, Denys, Rekesh, Dima, Ma, Ding, Pykhtar, Dmytro, Ahn, Dong, Riach, Duncan, Stosic, Dusan, Long, Eileen, Segal, Elad, Evans, Ellie, Chung, Eric, Galinkin, Erick, Bakhturina, Evelina, Dobrowolska, Ewa, Jia, Fei, Liu, Fuxiao, Prasad, Gargi, Shen, Gerald, Liu, Guilin, Chen, Guo, Qian, Haifeng, Ngo, Helen, Liu, Hongbin, Li, Hui, Gitman, Igor, Karmanov, Ilia, Moshkov, Ivan, Golan, Izik, Kautz, Jan, Scowcroft, Jane Polak, Casper, Jared, Seppanen, Jarno, Lu, Jason, Sewall, Jason, Zeng, Jiaqi, You, Jiaxuan, Zhang, Jimmy, Zhang, Jing, Huang, Jining, Xue, Jinze, Huang, Jocelyn, Conway, Joey, Kamalu, John, Barker, Jon, Cohen, Jonathan, Jennings, Joseph, Parmar, Jupinder, Sapra, Karan, Briski, Kari, Chumachenko, Kateryna, Luna, Katherine, Santhanam, Keshav, Kong, Kezhi, Sivamani, Kirthi, Pawelec, Krzysztof, Anik, Kumar, Li, Kunlun, McAfee, Lawrence, Derczynski, Leon, Pavao, Lindsey, Vega, Luis, Voegtle, Lukas, Bala, Maciej, de Melo, Maer Rodrigues, Sreedhar, Makesh Narsimhan, Chochowski, Marcin, Kliegl, Markus, Stepniewska-Dziubinska, Marta, Le, Matthieu, Novikov, Matvei, Samadi, Mehrzad, Andersch, Michael, Evans, Michael, Martinez, Miguel, Chrzanowski, Mike, Ranzinger, Mike, Blaz, Mikolaj, Smelyanskiy, Misha, Fawzy, Mohamed, Shoeybi, Mohammad, Patwary, Mostofa, Lee, Nayeon, Tajbakhsh, Nima, Xu, Ning, Rybakov, Oleg, Kuchaiev, Oleksii, Delalleau, Olivier, Nitski, Osvald, Chadha, Parth, Shamis, Pasha, Micikevicius, Paulius, Molchanov, Pavlo, Dykas, Peter, Fischer, Philipp, Aquilanti, Pierre-Yves, Bialecki, Piotr, Varshney, Prasoon, Gundecha, Pritam, Tredak, Przemek, Karimi, Rabeeh, Kandu, Rahul, El-Yaniv, Ran, Joshi, Raviraj, Waleffe, Roger, Zhang, Ruoxi, Kavanaugh, Sabrina, Jain, Sahil, Kriman, Samuel, Lym, Sangkug, Satheesh, Sanjeev, Muralidharan, Saurav, Narenthiran, Sean, Anandaraj, Selvaraj, Bak, Seonmyeong, Kashirsky, Sergey, Han, Seungju, Acharya, Shantanu, Ghosh, Shaona, Sreenivas, Sharath Turuvekere, Clay, Sharon, Thomas, Shelby, Prabhumoye, Shrimai, Pachori, Shubham, Toshniwal, Shubham, Prayaga, Shyamala, Jain, Siddhartha, Das, Sirshak, Kierat, Slawek, Majumdar, Somshubra, Han, Song, Singhal, Soumye, Niverty, Sriharsha, Alborghetti, Stefania, Panguluri, Suseella, Bhendigeri, Swetha, Akter, Syeda Nahida, Migacz, Szymon, Shiri, Tal, Kong, Terry, Roman, Timo, Ronen, Tomer, Saar, Trisha, Konuk, Tugrul, Rintamaki, Tuomas, Poon, Tyler, De, Ushnish, Noroozi, Vahid, Singh, Varun, Korthikanti, Vijay, Kurin, Vitaly, Ahmad, Wasi Uddin, Du, Wei, Ping, Wei, Dai, Wenliang, Byeon, Wonmin, Ren, Xiaowei, Xu, Yao, Choi, Yejin, Zhang, Yian, Lin, Ying, Suhara, Yoshi, Yu, Zhiding, Li, Zhiqi, Li, Zhiyu, Zhu, Zhongbo, Yang, Zhuolin, Chen, Zijia
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	https://arxiv.org/abs/2504.03624
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866908521732243456
author	NVIDIA : Blakeman, Aaron Basant, Aarti Khattar, Abhinav Renduchintala, Adithya Bercovich, Akhiad Ficek, Aleksander Bjorlin, Alexis Taghibakhshi, Ali Deshmukh, Amala Sanjay Mahabaleshwarkar, Ameya Sunil Tao, Andrew Shors, Anna Aithal, Ashwath Poojary, Ashwin Dattagupta, Ayush Buddharaju, Balaram Chen, Bobby Ginsburg, Boris Wang, Boxin Norick, Brandon Butterfield, Brian Catanzaro, Bryan del Mundo, Carlo Dong, Chengyu Harvey, Christine Parisien, Christopher Su, Dan Korzekwa, Daniel Yin, Danny Gitman, Daria Mosallanezhad, David Narayanan, Deepak Fridman, Denys Rekesh, Dima Ma, Ding Pykhtar, Dmytro Ahn, Dong Riach, Duncan Stosic, Dusan Long, Eileen Segal, Elad Evans, Ellie Chung, Eric Galinkin, Erick Bakhturina, Evelina Dobrowolska, Ewa Jia, Fei Liu, Fuxiao Prasad, Gargi Shen, Gerald Liu, Guilin Chen, Guo Qian, Haifeng Ngo, Helen Liu, Hongbin Li, Hui Gitman, Igor Karmanov, Ilia Moshkov, Ivan Golan, Izik Kautz, Jan Scowcroft, Jane Polak Casper, Jared Seppanen, Jarno Lu, Jason Sewall, Jason Zeng, Jiaqi You, Jiaxuan Zhang, Jimmy Zhang, Jing Huang, Jining Xue, Jinze Huang, Jocelyn Conway, Joey Kamalu, John Barker, Jon Cohen, Jonathan Jennings, Joseph Parmar, Jupinder Sapra, Karan Briski, Kari Chumachenko, Kateryna Luna, Katherine Santhanam, Keshav Kong, Kezhi Sivamani, Kirthi Pawelec, Krzysztof Anik, Kumar Li, Kunlun McAfee, Lawrence Derczynski, Leon Pavao, Lindsey Vega, Luis Voegtle, Lukas Bala, Maciej de Melo, Maer Rodrigues Sreedhar, Makesh Narsimhan Chochowski, Marcin Kliegl, Markus Stepniewska-Dziubinska, Marta Le, Matthieu Novikov, Matvei Samadi, Mehrzad Andersch, Michael Evans, Michael Martinez, Miguel Chrzanowski, Mike Ranzinger, Mike Blaz, Mikolaj Smelyanskiy, Misha Fawzy, Mohamed Shoeybi, Mohammad Patwary, Mostofa Lee, Nayeon Tajbakhsh, Nima Xu, Ning Rybakov, Oleg Kuchaiev, Oleksii Delalleau, Olivier Nitski, Osvald Chadha, Parth Shamis, Pasha Micikevicius, Paulius Molchanov, Pavlo Dykas, Peter Fischer, Philipp Aquilanti, Pierre-Yves Bialecki, Piotr Varshney, Prasoon Gundecha, Pritam Tredak, Przemek Karimi, Rabeeh Kandu, Rahul El-Yaniv, Ran Joshi, Raviraj Waleffe, Roger Zhang, Ruoxi Kavanaugh, Sabrina Jain, Sahil Kriman, Samuel Lym, Sangkug Satheesh, Sanjeev Muralidharan, Saurav Narenthiran, Sean Anandaraj, Selvaraj Bak, Seonmyeong Kashirsky, Sergey Han, Seungju Acharya, Shantanu Ghosh, Shaona Sreenivas, Sharath Turuvekere Clay, Sharon Thomas, Shelby Prabhumoye, Shrimai Pachori, Shubham Toshniwal, Shubham Prayaga, Shyamala Jain, Siddhartha Das, Sirshak Kierat, Slawek Majumdar, Somshubra Han, Song Singhal, Soumye Niverty, Sriharsha Alborghetti, Stefania Panguluri, Suseella Bhendigeri, Swetha Akter, Syeda Nahida Migacz, Szymon Shiri, Tal Kong, Terry Roman, Timo Ronen, Tomer Saar, Trisha Konuk, Tugrul Rintamaki, Tuomas Poon, Tyler De, Ushnish Noroozi, Vahid Singh, Varun Korthikanti, Vijay Kurin, Vitaly Ahmad, Wasi Uddin Du, Wei Ping, Wei Dai, Wenliang Byeon, Wonmin Ren, Xiaowei Xu, Yao Choi, Yejin Zhang, Yian Lin, Ying Suhara, Yoshi Yu, Zhiding Li, Zhiqi Li, Zhiyu Zhu, Zhongbo Yang, Zhuolin Chen, Zijia
author_facet	NVIDIA : Blakeman, Aaron Basant, Aarti Khattar, Abhinav Renduchintala, Adithya Bercovich, Akhiad Ficek, Aleksander Bjorlin, Alexis Taghibakhshi, Ali Deshmukh, Amala Sanjay Mahabaleshwarkar, Ameya Sunil Tao, Andrew Shors, Anna Aithal, Ashwath Poojary, Ashwin Dattagupta, Ayush Buddharaju, Balaram Chen, Bobby Ginsburg, Boris Wang, Boxin Norick, Brandon Butterfield, Brian Catanzaro, Bryan del Mundo, Carlo Dong, Chengyu Harvey, Christine Parisien, Christopher Su, Dan Korzekwa, Daniel Yin, Danny Gitman, Daria Mosallanezhad, David Narayanan, Deepak Fridman, Denys Rekesh, Dima Ma, Ding Pykhtar, Dmytro Ahn, Dong Riach, Duncan Stosic, Dusan Long, Eileen Segal, Elad Evans, Ellie Chung, Eric Galinkin, Erick Bakhturina, Evelina Dobrowolska, Ewa Jia, Fei Liu, Fuxiao Prasad, Gargi Shen, Gerald Liu, Guilin Chen, Guo Qian, Haifeng Ngo, Helen Liu, Hongbin Li, Hui Gitman, Igor Karmanov, Ilia Moshkov, Ivan Golan, Izik Kautz, Jan Scowcroft, Jane Polak Casper, Jared Seppanen, Jarno Lu, Jason Sewall, Jason Zeng, Jiaqi You, Jiaxuan Zhang, Jimmy Zhang, Jing Huang, Jining Xue, Jinze Huang, Jocelyn Conway, Joey Kamalu, John Barker, Jon Cohen, Jonathan Jennings, Joseph Parmar, Jupinder Sapra, Karan Briski, Kari Chumachenko, Kateryna Luna, Katherine Santhanam, Keshav Kong, Kezhi Sivamani, Kirthi Pawelec, Krzysztof Anik, Kumar Li, Kunlun McAfee, Lawrence Derczynski, Leon Pavao, Lindsey Vega, Luis Voegtle, Lukas Bala, Maciej de Melo, Maer Rodrigues Sreedhar, Makesh Narsimhan Chochowski, Marcin Kliegl, Markus Stepniewska-Dziubinska, Marta Le, Matthieu Novikov, Matvei Samadi, Mehrzad Andersch, Michael Evans, Michael Martinez, Miguel Chrzanowski, Mike Ranzinger, Mike Blaz, Mikolaj Smelyanskiy, Misha Fawzy, Mohamed Shoeybi, Mohammad Patwary, Mostofa Lee, Nayeon Tajbakhsh, Nima Xu, Ning Rybakov, Oleg Kuchaiev, Oleksii Delalleau, Olivier Nitski, Osvald Chadha, Parth Shamis, Pasha Micikevicius, Paulius Molchanov, Pavlo Dykas, Peter Fischer, Philipp Aquilanti, Pierre-Yves Bialecki, Piotr Varshney, Prasoon Gundecha, Pritam Tredak, Przemek Karimi, Rabeeh Kandu, Rahul El-Yaniv, Ran Joshi, Raviraj Waleffe, Roger Zhang, Ruoxi Kavanaugh, Sabrina Jain, Sahil Kriman, Samuel Lym, Sangkug Satheesh, Sanjeev Muralidharan, Saurav Narenthiran, Sean Anandaraj, Selvaraj Bak, Seonmyeong Kashirsky, Sergey Han, Seungju Acharya, Shantanu Ghosh, Shaona Sreenivas, Sharath Turuvekere Clay, Sharon Thomas, Shelby Prabhumoye, Shrimai Pachori, Shubham Toshniwal, Shubham Prayaga, Shyamala Jain, Siddhartha Das, Sirshak Kierat, Slawek Majumdar, Somshubra Han, Song Singhal, Soumye Niverty, Sriharsha Alborghetti, Stefania Panguluri, Suseella Bhendigeri, Swetha Akter, Syeda Nahida Migacz, Szymon Shiri, Tal Kong, Terry Roman, Timo Ronen, Tomer Saar, Trisha Konuk, Tugrul Rintamaki, Tuomas Poon, Tyler De, Ushnish Noroozi, Vahid Singh, Varun Korthikanti, Vijay Kurin, Vitaly Ahmad, Wasi Uddin Du, Wei Ping, Wei Dai, Wenliang Byeon, Wonmin Ren, Xiaowei Xu, Yao Choi, Yejin Zhang, Yian Lin, Ying Suhara, Yoshi Yu, Zhiding Li, Zhiqi Li, Zhiyu Zhu, Zhongbo Yang, Zhuolin Chen, Zijia
contents	As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3$\times$ faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.
format	Preprint
id	arxiv_https___arxiv_org_abs_2504_03624
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models NVIDIA : Blakeman, Aaron Basant, Aarti Khattar, Abhinav Renduchintala, Adithya Bercovich, Akhiad Ficek, Aleksander Bjorlin, Alexis Taghibakhshi, Ali Deshmukh, Amala Sanjay Mahabaleshwarkar, Ameya Sunil Tao, Andrew Shors, Anna Aithal, Ashwath Poojary, Ashwin Dattagupta, Ayush Buddharaju, Balaram Chen, Bobby Ginsburg, Boris Wang, Boxin Norick, Brandon Butterfield, Brian Catanzaro, Bryan del Mundo, Carlo Dong, Chengyu Harvey, Christine Parisien, Christopher Su, Dan Korzekwa, Daniel Yin, Danny Gitman, Daria Mosallanezhad, David Narayanan, Deepak Fridman, Denys Rekesh, Dima Ma, Ding Pykhtar, Dmytro Ahn, Dong Riach, Duncan Stosic, Dusan Long, Eileen Segal, Elad Evans, Ellie Chung, Eric Galinkin, Erick Bakhturina, Evelina Dobrowolska, Ewa Jia, Fei Liu, Fuxiao Prasad, Gargi Shen, Gerald Liu, Guilin Chen, Guo Qian, Haifeng Ngo, Helen Liu, Hongbin Li, Hui Gitman, Igor Karmanov, Ilia Moshkov, Ivan Golan, Izik Kautz, Jan Scowcroft, Jane Polak Casper, Jared Seppanen, Jarno Lu, Jason Sewall, Jason Zeng, Jiaqi You, Jiaxuan Zhang, Jimmy Zhang, Jing Huang, Jining Xue, Jinze Huang, Jocelyn Conway, Joey Kamalu, John Barker, Jon Cohen, Jonathan Jennings, Joseph Parmar, Jupinder Sapra, Karan Briski, Kari Chumachenko, Kateryna Luna, Katherine Santhanam, Keshav Kong, Kezhi Sivamani, Kirthi Pawelec, Krzysztof Anik, Kumar Li, Kunlun McAfee, Lawrence Derczynski, Leon Pavao, Lindsey Vega, Luis Voegtle, Lukas Bala, Maciej de Melo, Maer Rodrigues Sreedhar, Makesh Narsimhan Chochowski, Marcin Kliegl, Markus Stepniewska-Dziubinska, Marta Le, Matthieu Novikov, Matvei Samadi, Mehrzad Andersch, Michael Evans, Michael Martinez, Miguel Chrzanowski, Mike Ranzinger, Mike Blaz, Mikolaj Smelyanskiy, Misha Fawzy, Mohamed Shoeybi, Mohammad Patwary, Mostofa Lee, Nayeon Tajbakhsh, Nima Xu, Ning Rybakov, Oleg Kuchaiev, Oleksii Delalleau, Olivier Nitski, Osvald Chadha, Parth Shamis, Pasha Micikevicius, Paulius Molchanov, Pavlo Dykas, Peter Fischer, Philipp Aquilanti, Pierre-Yves Bialecki, Piotr Varshney, Prasoon Gundecha, Pritam Tredak, Przemek Karimi, Rabeeh Kandu, Rahul El-Yaniv, Ran Joshi, Raviraj Waleffe, Roger Zhang, Ruoxi Kavanaugh, Sabrina Jain, Sahil Kriman, Samuel Lym, Sangkug Satheesh, Sanjeev Muralidharan, Saurav Narenthiran, Sean Anandaraj, Selvaraj Bak, Seonmyeong Kashirsky, Sergey Han, Seungju Acharya, Shantanu Ghosh, Shaona Sreenivas, Sharath Turuvekere Clay, Sharon Thomas, Shelby Prabhumoye, Shrimai Pachori, Shubham Toshniwal, Shubham Prayaga, Shyamala Jain, Siddhartha Das, Sirshak Kierat, Slawek Majumdar, Somshubra Han, Song Singhal, Soumye Niverty, Sriharsha Alborghetti, Stefania Panguluri, Suseella Bhendigeri, Swetha Akter, Syeda Nahida Migacz, Szymon Shiri, Tal Kong, Terry Roman, Timo Ronen, Tomer Saar, Trisha Konuk, Tugrul Rintamaki, Tuomas Poon, Tyler De, Ushnish Noroozi, Vahid Singh, Varun Korthikanti, Vijay Kurin, Vitaly Ahmad, Wasi Uddin Du, Wei Ping, Wei Dai, Wenliang Byeon, Wonmin Ren, Xiaowei Xu, Yao Choi, Yejin Zhang, Yian Lin, Ying Suhara, Yoshi Yu, Zhiding Li, Zhiqi Li, Zhiyu Zhu, Zhongbo Yang, Zhuolin Chen, Zijia Computation and Language Artificial Intelligence Machine Learning As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3$\times$ faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.
title	Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
topic	Computation and Language Artificial Intelligence Machine Learning
url	https://arxiv.org/abs/2504.03624

Similar Items