Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.25149 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910041296076800 |
|---|---|
| author | NVIDIA Abecassis, Felix Agrusa, Anjulie Ahn, Dong Alben, Jonah Alborghetti, Stefania Andersch, Michael Arayandi, Sivakumar Bjorlin, Alexis Blakeman, Aaron Briones, Evan Buck, Ian Catanzaro, Bryan Chang, Muya Choi, Jinhang Chrzanowski, Mike Chung, Eric Cui, Victor Dai, Steve Rouhani, Bita Darvish del Mundo, Carlo Donia, Deena Eryilmaz, Burc Estela, Henry Goel, Abhinav Goncharov, Oleg Guvvala, Yugi Hesse, Robert Hewett, Russell Hum, Herbert Kapasi, Ujval Khailany, Brucek Khona, Mikail Knight, Nick Kondratenko, Alex Krashinsky, Ronny Lanir, Ben Layton, Simon Lightstone, Michael Lo, Daniel Micikevicius, Paulius Mishra, Asit Moon, Tim Narayanan, Deepak Ni, Chao Paithankar, Abhijit Pasumarthi, Satish Patel, Ankit Patwary, Mostofa Poojary, Ashwin Prasad, Gargi Priyadarshi, Sweta Qin, Yigong Ren, Xiaowei Rybakov, Oleg Sakr, Charbel Satheesh, Sanjeev Sergienko, Stas Shamis, Pasha Shankar, Kirthi Sharma, Nishant Shoeybi, Mohammad Siu, Michael Smelyanskiy, Misha Stosic, Darko Stosic, Dusan Su, Bor-Yiing Sun, Frank Tajbakhsh, Nima Thomas, Shelby Tredak, Przemek Tsykunov, Evgeny Vaithilingam, Gandhi Vavre, Aditya Venkatesan, Rangharajan Waleffe, Roger Wan, Qiyu Wang, Hexin Wang, Mengdi Wei, Lizzie Wu, Hao Wu, Evan Wyss, Keith Xu, Ning Xue, Jinze Yang, Charlene Zhai, Yujia Zhang, Ruoxi Zhu, Jingyang Zhu, Zhongbo |
| author_facet | NVIDIA Abecassis, Felix Agrusa, Anjulie Ahn, Dong Alben, Jonah Alborghetti, Stefania Andersch, Michael Arayandi, Sivakumar Bjorlin, Alexis Blakeman, Aaron Briones, Evan Buck, Ian Catanzaro, Bryan Chang, Muya Choi, Jinhang Chrzanowski, Mike Chung, Eric Cui, Victor Dai, Steve Rouhani, Bita Darvish del Mundo, Carlo Donia, Deena Eryilmaz, Burc Estela, Henry Goel, Abhinav Goncharov, Oleg Guvvala, Yugi Hesse, Robert Hewett, Russell Hum, Herbert Kapasi, Ujval Khailany, Brucek Khona, Mikail Knight, Nick Kondratenko, Alex Krashinsky, Ronny Lanir, Ben Layton, Simon Lightstone, Michael Lo, Daniel Micikevicius, Paulius Mishra, Asit Moon, Tim Narayanan, Deepak Ni, Chao Paithankar, Abhijit Pasumarthi, Satish Patel, Ankit Patwary, Mostofa Poojary, Ashwin Prasad, Gargi Priyadarshi, Sweta Qin, Yigong Ren, Xiaowei Rybakov, Oleg Sakr, Charbel Satheesh, Sanjeev Sergienko, Stas Shamis, Pasha Shankar, Kirthi Sharma, Nishant Shoeybi, Mohammad Siu, Michael Smelyanskiy, Misha Stosic, Darko Stosic, Dusan Su, Bor-Yiing Sun, Frank Tajbakhsh, Nima Thomas, Shelby Tredak, Przemek Tsykunov, Evgeny Vaithilingam, Gandhi Vavre, Aditya Venkatesan, Rangharajan Waleffe, Roger Wan, Qiyu Wang, Hexin Wang, Mengdi Wei, Lizzie Wu, Hao Wu, Evan Wyss, Keith Xu, Ning Xue, Jinze Yang, Charlene Zhai, Yujia Zhang, Ruoxi Zhu, Jingyang Zhu, Zhongbo |
| contents | Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons.
In this study, we introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format. Our method integrates Random Hadamard transforms (RHT) to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, utilizes stochastic rounding for unbiased gradient estimation, and incorporates selective high-precision layers. We validate our approach by training a 12-billion-parameter model on 10 trillion tokens -- the longest publicly documented training run in 4-bit precision to date. Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline. These findings highlight that NVFP4, when combined with our training approach, represents a major step forward in narrow-precision LLM training algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_25149 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pretraining Large Language Models with NVFP4 NVIDIA Abecassis, Felix Agrusa, Anjulie Ahn, Dong Alben, Jonah Alborghetti, Stefania Andersch, Michael Arayandi, Sivakumar Bjorlin, Alexis Blakeman, Aaron Briones, Evan Buck, Ian Catanzaro, Bryan Chang, Muya Choi, Jinhang Chrzanowski, Mike Chung, Eric Cui, Victor Dai, Steve Rouhani, Bita Darvish del Mundo, Carlo Donia, Deena Eryilmaz, Burc Estela, Henry Goel, Abhinav Goncharov, Oleg Guvvala, Yugi Hesse, Robert Hewett, Russell Hum, Herbert Kapasi, Ujval Khailany, Brucek Khona, Mikail Knight, Nick Kondratenko, Alex Krashinsky, Ronny Lanir, Ben Layton, Simon Lightstone, Michael Lo, Daniel Micikevicius, Paulius Mishra, Asit Moon, Tim Narayanan, Deepak Ni, Chao Paithankar, Abhijit Pasumarthi, Satish Patel, Ankit Patwary, Mostofa Poojary, Ashwin Prasad, Gargi Priyadarshi, Sweta Qin, Yigong Ren, Xiaowei Rybakov, Oleg Sakr, Charbel Satheesh, Sanjeev Sergienko, Stas Shamis, Pasha Shankar, Kirthi Sharma, Nishant Shoeybi, Mohammad Siu, Michael Smelyanskiy, Misha Stosic, Darko Stosic, Dusan Su, Bor-Yiing Sun, Frank Tajbakhsh, Nima Thomas, Shelby Tredak, Przemek Tsykunov, Evgeny Vaithilingam, Gandhi Vavre, Aditya Venkatesan, Rangharajan Waleffe, Roger Wan, Qiyu Wang, Hexin Wang, Mengdi Wei, Lizzie Wu, Hao Wu, Evan Wyss, Keith Xu, Ning Xue, Jinze Yang, Charlene Zhai, Yujia Zhang, Ruoxi Zhu, Jingyang Zhu, Zhongbo Computation and Language Artificial Intelligence Machine Learning Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons. In this study, we introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format. Our method integrates Random Hadamard transforms (RHT) to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, utilizes stochastic rounding for unbiased gradient estimation, and incorporates selective high-precision layers. We validate our approach by training a 12-billion-parameter model on 10 trillion tokens -- the longest publicly documented training run in 4-bit precision to date. Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline. These findings highlight that NVFP4, when combined with our training approach, represents a major step forward in narrow-precision LLM training algorithms. |
| title | Pretraining Large Language Models with NVFP4 |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.25149 |