AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2022
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917656005705728 |
|---|---|
| author | Yang, Enneng Pan, Junwei Wang, Ximei Yu, Haibin Shen, Li Chen, Xihua Xiao, Lei Jiang, Jie Guo, Guibing |
| author_facet | Yang, Enneng Pan, Junwei Wang, Ximei Yu, Haibin Shen, Li Chen, Xihua Xiao, Lei Jiang, Jie Guo, Guibing |
| contents | Multi-task learning (MTL) models have demonstrated impressive results in computer vision, natural language processing, and recommender systems. Even though many approaches have been proposed, how well these approaches balance different tasks on each parameter still remains unclear. In this paper, we propose to measure the task dominance degree of a parameter by the total updates of each task on this parameter. Specifically, we compute the total updates by the exponentially decaying Average of the squared Updates (AU) on a parameter from the corresponding task.Based on this novel metric, we observe that many parameters in existing MTL methods, especially those in the higher shared layers, are still dominated by one or several tasks. The dominance of AU is mainly due to the dominance of accumulative gradients from one or several tasks. Motivated by this, we propose a Task-wise Adaptive learning rate approach, AdaTask in short, to separate the \emph{accumulative gradients} and hence the learning rate of each task for each parameter in adaptive learning rate approaches (e.g., AdaGrad, RMSProp, and Adam). Comprehensive experiments on computer vision and recommender system MTL datasets demonstrate that AdaTask significantly improves the performance of dominated tasks, resulting SOTA average task-wise performance. Analysis on both synthetic and real-world datasets shows AdaTask balance parameters in every shared layer well. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_15055 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning Yang, Enneng Pan, Junwei Wang, Ximei Yu, Haibin Shen, Li Chen, Xihua Xiao, Lei Jiang, Jie Guo, Guibing Machine Learning Computer Vision and Pattern Recognition Information Retrieval Multi-task learning (MTL) models have demonstrated impressive results in computer vision, natural language processing, and recommender systems. Even though many approaches have been proposed, how well these approaches balance different tasks on each parameter still remains unclear. In this paper, we propose to measure the task dominance degree of a parameter by the total updates of each task on this parameter. Specifically, we compute the total updates by the exponentially decaying Average of the squared Updates (AU) on a parameter from the corresponding task.Based on this novel metric, we observe that many parameters in existing MTL methods, especially those in the higher shared layers, are still dominated by one or several tasks. The dominance of AU is mainly due to the dominance of accumulative gradients from one or several tasks. Motivated by this, we propose a Task-wise Adaptive learning rate approach, AdaTask in short, to separate the \emph{accumulative gradients} and hence the learning rate of each task for each parameter in adaptive learning rate approaches (e.g., AdaGrad, RMSProp, and Adam). Comprehensive experiments on computer vision and recommender system MTL datasets demonstrate that AdaTask significantly improves the performance of dominated tasks, resulting SOTA average task-wise performance. Analysis on both synthetic and real-world datasets shows AdaTask balance parameters in every shared layer well. |
| title | AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning |
| topic | Machine Learning Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2211.15055 |