Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2101.09868 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912176512434176 |
|---|---|
| author | Fu, Yonggan Guo, Han Li, Meng Yang, Xin Ding, Yining Chandra, Vikas Lin, Yingyan Celine |
| author_facet | Fu, Yonggan Guo, Han Li, Meng Yang, Xin Ding, Yining Chandra, Vikas Lin, Yingyan Celine |
| contents | Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at: https://github.com/RICE-EIC/CPT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2101_09868 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | CPT: Efficient Deep Neural Network Training via Cyclic Precision Fu, Yonggan Guo, Han Li, Meng Yang, Xin Ding, Yining Chandra, Vikas Lin, Yingyan Celine Machine Learning Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at: https://github.com/RICE-EIC/CPT. |
| title | CPT: Efficient Deep Neural Network Training via Cyclic Precision |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2101.09868 |