Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2211.10737 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916267004264448 |
|---|---|
| author | Harma, Simla Burcu Chakraborty, Ayan Sperry, Nicholas Falsafi, Babak Jaggi, Martin Oh, Yunho |
| author_facet | Harma, Simla Burcu Chakraborty, Ayan Sperry, Nicholas Falsafi, Babak Jaggi, Martin Oh, Yunho |
| contents | The unprecedented demand for computing resources to train DNN models has led to a search for minimal numerical encoding. Recent state-of-the-art (SOTA) proposals advocate for multi-level scaled narrow bitwidth numerical formats. In this paper, we show that single-level scaling is sufficient to maintain training accuracy while maximizing arithmetic density. We identify a previously proposed single-level scaled format for 8-bit training, Hybrid Block Floating Point (HBFP), as the optimal candidate to minimize. We perform a full-scale exploration of the HBFP design space using mathematical tools to study the interplay among various parameters and identify opportunities for even smaller encodings across layers and epochs. Based on our findings, we propose Accuracy Booster, a mixed-mantissa HBFP technique that uses 4-bit mantissas for over 99% of all arithmetic operations in training and 6-bit mantissas only in the last epoch and first/last layers. We show Accuracy Booster enables increasing arithmetic density over all other SOTA formats by at least 2.3x while achieving state-of-the-art accuracies in 4-bit training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_10737 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training Harma, Simla Burcu Chakraborty, Ayan Sperry, Nicholas Falsafi, Babak Jaggi, Martin Oh, Yunho Machine Learning The unprecedented demand for computing resources to train DNN models has led to a search for minimal numerical encoding. Recent state-of-the-art (SOTA) proposals advocate for multi-level scaled narrow bitwidth numerical formats. In this paper, we show that single-level scaling is sufficient to maintain training accuracy while maximizing arithmetic density. We identify a previously proposed single-level scaled format for 8-bit training, Hybrid Block Floating Point (HBFP), as the optimal candidate to minimize. We perform a full-scale exploration of the HBFP design space using mathematical tools to study the interplay among various parameters and identify opportunities for even smaller encodings across layers and epochs. Based on our findings, we propose Accuracy Booster, a mixed-mantissa HBFP technique that uses 4-bit mantissas for over 99% of all arithmetic operations in training and 6-bit mantissas only in the last epoch and first/last layers. We show Accuracy Booster enables increasing arithmetic density over all other SOTA formats by at least 2.3x while achieving state-of-the-art accuracies in 4-bit training. |
| title | Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2211.10737 |