Improving Language Plasticity via Pretraining with Active Forgetting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929207338074112 |
|---|---|
| author | Chen, Yihong Marchisio, Kelly Raileanu, Roberta Adelani, David Ifeoluwa Stenetorp, Pontus Riedel, Sebastian Artetxe, Mikel |
| author_facet | Chen, Yihong Marchisio, Kelly Raileanu, Roberta Adelani, David Ifeoluwa Stenetorp, Pontus Riedel, Sebastian Artetxe, Mikel |
| contents | Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within a limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation but also outperform standard ones in a low-data regime, particularly for languages that are distant from English. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_01163 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Improving Language Plasticity via Pretraining with Active Forgetting Chen, Yihong Marchisio, Kelly Raileanu, Roberta Adelani, David Ifeoluwa Stenetorp, Pontus Riedel, Sebastian Artetxe, Mikel Computation and Language Machine Learning Neural and Evolutionary Computing Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within a limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation but also outperform standard ones in a low-data regime, particularly for languages that are distant from English. |
| title | Improving Language Plasticity via Pretraining with Active Forgetting |
| topic | Computation and Language Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2307.01163 |