Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.12567 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911791130345472 |
|---|---|
| author | Gong, Linyuan Xiong, Chenyan Liu, Xiaodong Bajaj, Payal Xie, Yiqing Cheung, Alvin Gao, Jianfeng Song, Xia |
| author_facet | Gong, Linyuan Xiong, Chenyan Liu, Xiaodong Bajaj, Payal Xie, Yiqing Cheung, Alvin Gao, Jianfeng Song, Xia |
| contents | This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0-11B model with only 8% of its parameters. Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https://github.com/gonglinyuan/metro_t0. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_12567 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers Gong, Linyuan Xiong, Chenyan Liu, Xiaodong Bajaj, Payal Xie, Yiqing Cheung, Alvin Gao, Jianfeng Song, Xia Computation and Language This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0-11B model with only 8% of its parameters. Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https://github.com/gonglinyuan/metro_t0. |
| title | Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2305.12567 |