Dual-Balancing for Multi-Task Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911286768435200 |
|---|---|
| author | Lin, Baijiong Jiang, Weisen Ye, Feiyang Zhang, Yu Chen, Pengguang Chen, Ying-Cong Liu, Shu Tsang, Ivor W. Kwok, James T. |
| author_facet | Lin, Baijiong Jiang, Weisen Ye, Feiyang Zhang, Yu Chen, Pengguang Chen, Ying-Cong Liu, Shu Tsang, Ivor W. Kwok, James T. |
| contents | Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_12029 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Dual-Balancing for Multi-Task Learning Lin, Baijiong Jiang, Weisen Ye, Feiyang Zhang, Yu Chen, Pengguang Chen, Ying-Cong Liu, Shu Tsang, Ivor W. Kwok, James T. Machine Learning Artificial Intelligence Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art. |
| title | Dual-Balancing for Multi-Task Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2308.12029 |