Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2302.02060 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913266854264832 |
|---|---|
| author | Meng, Yu Krishnan, Jitin Wang, Sinong Wang, Qifan Mao, Yuning Fang, Han Ghazvininejad, Marjan Han, Jiawei Zettlemoyer, Luke |
| author_facet | Meng, Yu Krishnan, Jitin Wang, Sinong Wang, Qifan Mao, Yuning Fang, Han Ghazvininejad, Marjan Han, Jiawei Zettlemoyer, Luke |
| contents | Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens. Motivated by the identified issue, we propose MAE-LM, which pretrains the Masked Autoencoder architecture with MLM where $\texttt{[MASK]}$ tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM-pretrained models across different pretraining settings and model sizes when fine-tuned on the GLUE and SQuAD benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_02060 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Representation Deficiency in Masked Language Modeling Meng, Yu Krishnan, Jitin Wang, Sinong Wang, Qifan Mao, Yuning Fang, Han Ghazvininejad, Marjan Han, Jiawei Zettlemoyer, Luke Computation and Language Machine Learning Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens. Motivated by the identified issue, we propose MAE-LM, which pretrains the Masked Autoencoder architecture with MLM where $\texttt{[MASK]}$ tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM-pretrained models across different pretraining settings and model sizes when fine-tuned on the GLUE and SQuAD benchmarks. |
| title | Representation Deficiency in Masked Language Modeling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2302.02060 |