Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.00454 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929501170040832 |
|---|---|
| author | Ri, Ryokan Kiyono, Shun Takase, Sho |
| author_facet | Ri, Ryokan Kiyono, Shun Takase, Sho |
| contents | Zero-shot cross-lingual transfer by fine-tuning multilingual pretrained models shows promise for low-resource languages, but often suffers from misalignment of internal representations between languages. We hypothesize that even when the model cannot generalize across languages effectively in fine-tuning, it still captures cross-lingual correspondence useful for cross-lingual transfer. We explore this hypothesis with Self-Translate-Train, a method that lets large language models (LLMs) to translate training data into the target language and fine-tunes the model on its own generated data. By demonstrating that Self-Translate-Train outperforms zero-shot transfer, we encourage further exploration of better methods to elicit cross-lingual capabilities of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00454 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability Ri, Ryokan Kiyono, Shun Takase, Sho Computation and Language Zero-shot cross-lingual transfer by fine-tuning multilingual pretrained models shows promise for low-resource languages, but often suffers from misalignment of internal representations between languages. We hypothesize that even when the model cannot generalize across languages effectively in fine-tuning, it still captures cross-lingual correspondence useful for cross-lingual transfer. We explore this hypothesis with Self-Translate-Train, a method that lets large language models (LLMs) to translate training data into the target language and fine-tunes the model on its own generated data. By demonstrating that Self-Translate-Train outperforms zero-shot transfer, we encourage further exploration of better methods to elicit cross-lingual capabilities of LLMs. |
| title | Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.00454 |