Gradient Deconfliction via Orthogonal Projections onto Subspaces For Multi-task Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916644397252608 |
|---|---|
| author | Zhu, Shijie Zhao, Hui Wu, Tianshu Wang, Pengjie Deng, Hongbo Xu, Jian Zheng, Bo |
| author_facet | Zhu, Shijie Zhao, Hui Wu, Tianshu Wang, Pengjie Deng, Hongbo Xu, Jian Zheng, Bo |
| contents | Although multi-task learning (MTL) has been a preferred approach and successfully applied in many real-world scenarios, MTL models are not guaranteed to outperform single-task models on all tasks mainly due to the negative effects of conflicting gradients among the tasks. In this paper, we fully examine the influence of conflicting gradients and further emphasize the importance and advantages of achieving non-conflicting gradients which allows simple but effective trade-off strategies among the tasks with stable performance. Based on our findings, we propose the Gradient Deconfliction via Orthogonal Projections onto Subspaces (GradOPS) spanned by other task-specific gradients. Our method not only solves all conflicts among the tasks, but can also effectively search for diverse solutions towards different trade-off preferences among the tasks. Theoretical analysis on convergence is provided, and performance of our algorithm is fully testified on multiple benchmarks in various domains. Results demonstrate that our method can effectively find multiple state-of-the-art solutions with different trade-off strategies among the tasks on multiple datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_03438 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gradient Deconfliction via Orthogonal Projections onto Subspaces For Multi-task Learning Zhu, Shijie Zhao, Hui Wu, Tianshu Wang, Pengjie Deng, Hongbo Xu, Jian Zheng, Bo Machine Learning Although multi-task learning (MTL) has been a preferred approach and successfully applied in many real-world scenarios, MTL models are not guaranteed to outperform single-task models on all tasks mainly due to the negative effects of conflicting gradients among the tasks. In this paper, we fully examine the influence of conflicting gradients and further emphasize the importance and advantages of achieving non-conflicting gradients which allows simple but effective trade-off strategies among the tasks with stable performance. Based on our findings, we propose the Gradient Deconfliction via Orthogonal Projections onto Subspaces (GradOPS) spanned by other task-specific gradients. Our method not only solves all conflicts among the tasks, but can also effectively search for diverse solutions towards different trade-off preferences among the tasks. Theoretical analysis on convergence is provided, and performance of our algorithm is fully testified on multiple benchmarks in various domains. Results demonstrate that our method can effectively find multiple state-of-the-art solutions with different trade-off strategies among the tasks on multiple datasets. |
| title | Gradient Deconfliction via Orthogonal Projections onto Subspaces For Multi-task Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.03438 |