Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929536571015168 |
|---|---|
| author | Chronopoulou, Alexandra Pfeiffer, Jonas Maynez, Joshua Wang, Xinyi Ruder, Sebastian Agrawal, Priyanka |
| author_facet | Chronopoulou, Alexandra Pfeiffer, Jonas Maynez, Joshua Wang, Xinyi Ruder, Sebastian Agrawal, Priyanka |
| contents | Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09344 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization Chronopoulou, Alexandra Pfeiffer, Jonas Maynez, Joshua Wang, Xinyi Ruder, Sebastian Agrawal, Priyanka Computation and Language Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules. |
| title | Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09344 |