Faster Inference of LLMs using FP8 on the Intel Gaudi
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916653312245760 |
|---|---|
| author | Lee, Joonhyung Markovich-Golan, Shmulik Ohayon, Daniel Hanani, Yair Park, Gunho Kim, Byeongwook Karnieli, Asaf Livne, Uri Shen, Haihao Huang, Tai Kwon, Se Jung Lee, Dongsoo |
| author_facet | Lee, Joonhyung Markovich-Golan, Shmulik Ohayon, Daniel Hanani, Yair Park, Gunho Kim, Byeongwook Karnieli, Asaf Livne, Uri Shen, Haihao Huang, Tai Kwon, Se Jung Lee, Dongsoo |
| contents | Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting available hardware resources. Despite the incorporation of FP8 in commercially available neural network accelerators, a comprehensive exposition of its underlying mechanisms, along with rigorous performance and accuracy evaluations, is still lacking. In this work, we contribute in three significant ways. First, we analyze the implementation details and quantization options associated with FP8 for inference on the Intel Gaudi AI accelerator. Second, we empirically quantify the throughput improvements afforded by the use of FP8 at both the operator level and in end-to-end scenarios. Third, we assess the accuracy impact of various FP8 quantization methods. Our experimental results indicate that the Intel Gaudi 2 accelerator consistently achieves high computational unit utilization, frequently exceeding 90% MFU, while incurring an accuracy degradation of less than 1%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09975 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Faster Inference of LLMs using FP8 on the Intel Gaudi Lee, Joonhyung Markovich-Golan, Shmulik Ohayon, Daniel Hanani, Yair Park, Gunho Kim, Byeongwook Karnieli, Asaf Livne, Uri Shen, Haihao Huang, Tai Kwon, Se Jung Lee, Dongsoo Hardware Architecture Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting available hardware resources. Despite the incorporation of FP8 in commercially available neural network accelerators, a comprehensive exposition of its underlying mechanisms, along with rigorous performance and accuracy evaluations, is still lacking. In this work, we contribute in three significant ways. First, we analyze the implementation details and quantization options associated with FP8 for inference on the Intel Gaudi AI accelerator. Second, we empirically quantify the throughput improvements afforded by the use of FP8 at both the operator level and in end-to-end scenarios. Third, we assess the accuracy impact of various FP8 quantization methods. Our experimental results indicate that the Intel Gaudi 2 accelerator consistently achieves high computational unit utilization, frequently exceeding 90% MFU, while incurring an accuracy degradation of less than 1%. |
| title | Faster Inference of LLMs using FP8 on the Intel Gaudi |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2503.09975 |