The Hidden Power of Pure 16-bit Floating-Point Neural Networks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914781966893056 |
|---|---|
| author | Yun, Juyoung Kang, Byungkon Fu, Zhoulai |
| author_facet | Yun, Juyoung Kang, Byungkon Fu, Zhoulai |
| contents | Lowering the precision of neural networks from the prevalent 32-bit precision has long been considered harmful to performance, despite the gain in space and time. Many works propose various techniques to implement half-precision neural networks, but none study pure 16-bit settings. This paper investigates the unexpected performance gain of pure 16-bit neural networks over the 32-bit networks in classification tasks. We present extensive experimental results that favorably compare various 16-bit neural networks' performance to those of the 32-bit models. In addition, a theoretical analysis of the efficiency of 16-bit models is provided, which is coupled with empirical evidence to back it up. Finally, we discuss situations in which low-precision training is indeed detrimental. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2301_12809 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | The Hidden Power of Pure 16-bit Floating-Point Neural Networks Yun, Juyoung Kang, Byungkon Fu, Zhoulai Machine Learning Artificial Intelligence Performance Lowering the precision of neural networks from the prevalent 32-bit precision has long been considered harmful to performance, despite the gain in space and time. Many works propose various techniques to implement half-precision neural networks, but none study pure 16-bit settings. This paper investigates the unexpected performance gain of pure 16-bit neural networks over the 32-bit networks in classification tasks. We present extensive experimental results that favorably compare various 16-bit neural networks' performance to those of the 32-bit models. In addition, a theoretical analysis of the efficiency of 16-bit models is provided, which is coupled with empirical evidence to back it up. Finally, we discuss situations in which low-precision training is indeed detrimental. |
| title | The Hidden Power of Pure 16-bit Floating-Point Neural Networks |
| topic | Machine Learning Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2301.12809 |