Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916358764101632 |
|---|---|
| author | Kirchdorfer, Lukas Elich, Cathrin Kutsche, Simon Stuckenschmidt, Heiner Schott, Lukas Köhler, Jan M. |
| author_facet | Kirchdorfer, Lukas Elich, Cathrin Kutsche, Simon Stuckenschmidt, Heiner Schott, Lukas Köhler, Jan M. |
| contents | With the rise of neural networks in various domains, multi-task learning (MTL) gained significant relevance. A key challenge in MTL is balancing individual task losses during neural network training to improve performance and efficiency through knowledge sharing across tasks. To address these challenges, we propose a novel task-weighting method by building on the most prevalent approach of Uncertainty Weighting and computing analytically optimal uncertainty-based weights, normalized by a softmax function with tunable temperature. Our approach yields comparable results to the combinatorially prohibitive, brute-force approach of Scalarization while offering a more cost-effective yet high-performing alternative. We conduct an extensive benchmark on various datasets and architectures. Our method consistently outperforms six other common weighting methods. Furthermore, we report noteworthy experimental findings for the practical application of MTL. For example, larger networks diminish the influence of weighting methods, and tuning the weight decay has a low impact compared to the learning rate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07985 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning Kirchdorfer, Lukas Elich, Cathrin Kutsche, Simon Stuckenschmidt, Heiner Schott, Lukas Köhler, Jan M. Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition With the rise of neural networks in various domains, multi-task learning (MTL) gained significant relevance. A key challenge in MTL is balancing individual task losses during neural network training to improve performance and efficiency through knowledge sharing across tasks. To address these challenges, we propose a novel task-weighting method by building on the most prevalent approach of Uncertainty Weighting and computing analytically optimal uncertainty-based weights, normalized by a softmax function with tunable temperature. Our approach yields comparable results to the combinatorially prohibitive, brute-force approach of Scalarization while offering a more cost-effective yet high-performing alternative. We conduct an extensive benchmark on various datasets and architectures. Our method consistently outperforms six other common weighting methods. Furthermore, we report noteworthy experimental findings for the practical application of MTL. For example, larger networks diminish the influence of weighting methods, and tuning the weight decay has a low impact compared to the learning rate. |
| title | Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.07985 |