Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912951827431424 |
|---|---|
| author | Chen, Xuxing He, Yun Xu, Jiayi Huang, Minhui Liu, Xiaoyi Liu, Boyang Tian, Fei Wei, Xiaohan Jin, Rong Park, Sem Long, Bo Feng, Xue |
| author_facet | Chen, Xuxing He, Yun Xu, Jiayi Huang, Minhui Liu, Xiaoyi Liu, Boyang Tian, Fei Wei, Xiaohan Jin, Rong Park, Sem Long, Bo Feng, Xue |
| contents | In machine learning, the goal of multi-task learning (MTL) is to optimize multiple objectives together. Recent works, for example, Multiple Gradient Descent Algorithm (MGDA) and its variants, show promising results with dynamically adjusted weights for different tasks to mitigate conflicts that may potentially degrade the performance on certain tasks. Despite the empirical success of MGDA-type methods, one major limitation of such methods is their computational inefficiency, as they require access to all task gradients. In this paper we introduce MARIGOLD, a unified algorithmic framework for efficiently solving MTL problems. Our method reveals that multi-task gradient balancing methods have a hierarchical structure, in which the model training and the gradient balancing are coupled during the whole optimization process and can be viewed as a bi-level optimization problem. Moreover, we showcase that the bi-level problem can be solved efficiently by leveraging zeroth-order method. Extensive experiments on both public datasets and industrial-scale datasets demonstrate the efficiency and superiority of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07389 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization Chen, Xuxing He, Yun Xu, Jiayi Huang, Minhui Liu, Xiaoyi Liu, Boyang Tian, Fei Wei, Xiaohan Jin, Rong Park, Sem Long, Bo Feng, Xue Machine Learning Optimization and Control In machine learning, the goal of multi-task learning (MTL) is to optimize multiple objectives together. Recent works, for example, Multiple Gradient Descent Algorithm (MGDA) and its variants, show promising results with dynamically adjusted weights for different tasks to mitigate conflicts that may potentially degrade the performance on certain tasks. Despite the empirical success of MGDA-type methods, one major limitation of such methods is their computational inefficiency, as they require access to all task gradients. In this paper we introduce MARIGOLD, a unified algorithmic framework for efficiently solving MTL problems. Our method reveals that multi-task gradient balancing methods have a hierarchical structure, in which the model training and the gradient balancing are coupled during the whole optimization process and can be viewed as a bi-level optimization problem. Moreover, we showcase that the bi-level problem can be solved efficiently by leveraging zeroth-order method. Extensive experiments on both public datasets and industrial-scale datasets demonstrate the efficiency and superiority of our method. |
| title | Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2603.07389 |