InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908598383149056 |
|---|---|
| author | Wang, Wenjun Cai, Shuo Xie, Congkai Feng, Mingfa Zhang, Yiming Li, Zhen Yang, Kejing Li, Ming Cao, Jiannong Yang, Hongxia |
| author_facet | Wang, Wenjun Cai, Shuo Xie, Congkai Feng, Mingfa Zhang, Yiming Li, Zhen Yang, Kejing Li, Ming Cao, Jiannong Yang, Hongxia |
| contents | The immense computational cost of training Large Language Models (LLMs) presents a major barrier to innovation. While FP8 training offers a promising solution with significant theoretical efficiency gains, its widespread adoption has been hindered by the lack of a comprehensive, open-source training recipe. To bridge this gap, we introduce an end-to-end FP8 training recipe that seamlessly integrates continual pre-training and supervised fine-tuning. Our methodology employs a fine-grained, hybrid-granularity quantization strategy to maintain numerical fidelity while maximizing computational efficiency. Through extensive experiments, including the continue pre-training of models on a 160B-token corpus, we demonstrate that our recipe is not only remarkably stable but also essentially lossless, achieving performance on par with the BF16 baseline across a suite of reasoning benchmarks. Crucially, this is achieved with substantial efficiency improvements, including up to a 22% reduction in training time, a 14% decrease in peak memory usage, and a 19% increase in throughput. Our results establish FP8 as a practical and robust alternative to BF16, and we will release the accompanying code to further democratize large-scale model training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22536 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models Wang, Wenjun Cai, Shuo Xie, Congkai Feng, Mingfa Zhang, Yiming Li, Zhen Yang, Kejing Li, Ming Cao, Jiannong Yang, Hongxia Computation and Language Artificial Intelligence The immense computational cost of training Large Language Models (LLMs) presents a major barrier to innovation. While FP8 training offers a promising solution with significant theoretical efficiency gains, its widespread adoption has been hindered by the lack of a comprehensive, open-source training recipe. To bridge this gap, we introduce an end-to-end FP8 training recipe that seamlessly integrates continual pre-training and supervised fine-tuning. Our methodology employs a fine-grained, hybrid-granularity quantization strategy to maintain numerical fidelity while maximizing computational efficiency. Through extensive experiments, including the continue pre-training of models on a 160B-token corpus, we demonstrate that our recipe is not only remarkably stable but also essentially lossless, achieving performance on par with the BF16 baseline across a suite of reasoning benchmarks. Crucially, this is achieved with substantial efficiency improvements, including up to a 22% reduction in training time, a 14% decrease in peak memory usage, and a 19% increase in throughput. Our results establish FP8 as a practical and robust alternative to BF16, and we will release the accompanying code to further democratize large-scale model training. |
| title | InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.22536 |