MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909560562778112 |
|---|---|
| author | Yang, Yaming Muhtar, Dilxat Shen, Yelong Zhan, Yuefeng Liu, Jianfeng Wang, Yujing Sun, Hao Deng, Denvy Sun, Feng Zhang, Qi Chen, Weizhu Tong, Yunhai |
| author_facet | Yang, Yaming Muhtar, Dilxat Shen, Yelong Zhan, Yuefeng Liu, Jianfeng Wang, Yujing Sun, Hao Deng, Denvy Sun, Feng Zhang, Qi Chen, Weizhu Tong, Yunhai |
| contents | Parameter-efficient fine-tuning (PEFT) has been widely employed for domain adaptation, with LoRA being one of the most prominent methods due to its simplicity and effectiveness. However, in multi-task learning (MTL) scenarios, LoRA tends to obscure the distinction between tasks by projecting sparse high-dimensional features from different tasks into the same dense low-dimensional intrinsic space. This leads to task interference and suboptimal performance for LoRA and its variants. To tackle this challenge, we propose MTL-LoRA, which retains the advantages of low-rank adaptation while significantly enhancing MTL capabilities. MTL-LoRA augments LoRA by incorporating additional task-adaptive parameters that differentiate task-specific information and capture shared knowledge across various tasks within low-dimensional spaces. This approach enables pre-trained models to jointly adapt to different target domains with a limited number of trainable parameters. Comprehensive experimental results, including evaluations on public academic benchmarks for natural language understanding, commonsense reasoning, and image-text understanding, as well as real-world industrial text Ads relevance datasets, demonstrate that MTL-LoRA outperforms LoRA and its various variants with comparable or even fewer learnable parameters in MTL setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09437 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning Yang, Yaming Muhtar, Dilxat Shen, Yelong Zhan, Yuefeng Liu, Jianfeng Wang, Yujing Sun, Hao Deng, Denvy Sun, Feng Zhang, Qi Chen, Weizhu Tong, Yunhai Machine Learning Artificial Intelligence Computation and Language Parameter-efficient fine-tuning (PEFT) has been widely employed for domain adaptation, with LoRA being one of the most prominent methods due to its simplicity and effectiveness. However, in multi-task learning (MTL) scenarios, LoRA tends to obscure the distinction between tasks by projecting sparse high-dimensional features from different tasks into the same dense low-dimensional intrinsic space. This leads to task interference and suboptimal performance for LoRA and its variants. To tackle this challenge, we propose MTL-LoRA, which retains the advantages of low-rank adaptation while significantly enhancing MTL capabilities. MTL-LoRA augments LoRA by incorporating additional task-adaptive parameters that differentiate task-specific information and capture shared knowledge across various tasks within low-dimensional spaces. This approach enables pre-trained models to jointly adapt to different target domains with a limited number of trainable parameters. Comprehensive experimental results, including evaluations on public academic benchmarks for natural language understanding, commonsense reasoning, and image-text understanding, as well as real-world industrial text Ads relevance datasets, demonstrate that MTL-LoRA outperforms LoRA and its various variants with comparable or even fewer learnable parameters in MTL setting. |
| title | MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.09437 |