Dynamic Masking Rate Schedules for MLM Pretraining
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916120302190592 |
|---|---|
| author | Ankner, Zachary Saphra, Naomi Blalock, Davis Frankle, Jonathan Leavitt, Matthew L. |
| author_facet | Ankner, Zachary Saphra, Naomi Blalock, Davis Frankle, Jonathan Leavitt, Matthew L. |
| contents | Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout training. We find that linearly decreasing the masking rate over the course of pretraining improves average GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT-large, respectively, compared to fixed rate baselines. These gains come from exposure to both high and low masking rate regimes, providing benefits from both settings. Our results demonstrate that masking rate scheduling is a simple way to improve the quality of masked language models, achieving up to a 1.89x speedup in pretraining for BERT-base as well as a Pareto improvement for BERT-large. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15096 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Dynamic Masking Rate Schedules for MLM Pretraining Ankner, Zachary Saphra, Naomi Blalock, Davis Frankle, Jonathan Leavitt, Matthew L. Computation and Language Artificial Intelligence Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout training. We find that linearly decreasing the masking rate over the course of pretraining improves average GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT-large, respectively, compared to fixed rate baselines. These gains come from exposure to both high and low masking rate regimes, providing benefits from both settings. Our results demonstrate that masking rate scheduling is a simple way to improve the quality of masked language models, achieving up to a 1.89x speedup in pretraining for BERT-base as well as a Pareto improvement for BERT-large. |
| title | Dynamic Masking Rate Schedules for MLM Pretraining |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2305.15096 |