Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.17934 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911974302941184 |
|---|---|
| author | Lin, Chu-Cheng Wang, Xinyi Clark, Jonathan H. Lu, Han Zhu, Yun Whitehouse, Chenxi Yu, Hongkun |
| author_facet | Lin, Chu-Cheng Wang, Xinyi Clark, Jonathan H. Lu, Han Zhu, Yun Whitehouse, Chenxi Yu, Hongkun |
| contents | Adapting pretrained large language models (LLMs) to various downstream tasks in tens or hundreds of human languages is computationally expensive. Parameter-efficient fine-tuning (PEFT) significantly reduces the adaptation cost, by tuning only a small amount of parameters. However, common PEFT methods LoRA (Hu et al., 2022) suffer from suboptimal performance on diverse dataset mixtures, due to aggressive parameter tying and negative interference among different datasets. In this work, we propose Featurized Low-rank Mixtures (FLix), a novel PEFT method designed for effective multitask multilingual adaptation. FLix associates each unique dataset feature, such as the dataset's language or task, with its own low-rank weight update parameters. By composing feature-specific parameters for each dataset, FLix can accommodate diverse dataset mixtures and generalize better to unseen datasets. Our experiments show that FLix leads to significant improvements over a variety of tasks for both supervised learning and zero-shot settings with gains of up to $14.2$ inexact match points in zero-shot semantic parsing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17934 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixtures Lin, Chu-Cheng Wang, Xinyi Clark, Jonathan H. Lu, Han Zhu, Yun Whitehouse, Chenxi Yu, Hongkun Computation and Language Artificial Intelligence Adapting pretrained large language models (LLMs) to various downstream tasks in tens or hundreds of human languages is computationally expensive. Parameter-efficient fine-tuning (PEFT) significantly reduces the adaptation cost, by tuning only a small amount of parameters. However, common PEFT methods LoRA (Hu et al., 2022) suffer from suboptimal performance on diverse dataset mixtures, due to aggressive parameter tying and negative interference among different datasets. In this work, we propose Featurized Low-rank Mixtures (FLix), a novel PEFT method designed for effective multitask multilingual adaptation. FLix associates each unique dataset feature, such as the dataset's language or task, with its own low-rank weight update parameters. By composing feature-specific parameters for each dataset, FLix can accommodate diverse dataset mixtures and generalize better to unseen datasets. Our experiments show that FLix leads to significant improvements over a variety of tasks for both supervised learning and zero-shot settings with gains of up to $14.2$ inexact match points in zero-shot semantic parsing. |
| title | Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixtures |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.17934 |