SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929221296717824 |
|---|---|
| author | Ye, Ke Jiang, Heinrich Rostamizadeh, Afshin Chakrabarti, Ayan DeSalvo, Giulia Kagy, Jean-François Karydas, Lazaros Citovsky, Gui Kumar, Sanjiv |
| author_facet | Ye, Ke Jiang, Heinrich Rostamizadeh, Afshin Chakrabarti, Ayan DeSalvo, Giulia Kagy, Jean-François Karydas, Lazaros Citovsky, Gui Kumar, Sanjiv |
| contents | Pre-training large language models is known to be extremely resource intensive and often times inefficient, under-utilizing the information encapsulated in the training text sequences. In this paper, we present SpacTor, a new training procedure consisting of (1) a hybrid objective combining span corruption (SC) and token replacement detection (RTD), and (2) a two-stage curriculum that optimizes the hybrid objective over the initial $τ$ iterations, then transitions to standard SC loss. We show empirically that the effectiveness of the hybrid objective is tied to the two-stage pre-training schedule, and provide extensive analysis on why this is the case. In our experiments with encoder-decoder architectures (T5) on a variety of NLP tasks, SpacTor-T5 yields the same downstream performance as standard SC pre-training, while enabling a 50% reduction in pre-training iterations and 40% reduction in total FLOPs. Alternatively, given the same amount of computing budget, we find that SpacTor results in significantly improved downstream benchmark performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_13160 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection Ye, Ke Jiang, Heinrich Rostamizadeh, Afshin Chakrabarti, Ayan DeSalvo, Giulia Kagy, Jean-François Karydas, Lazaros Citovsky, Gui Kumar, Sanjiv Machine Learning Computation and Language Pre-training large language models is known to be extremely resource intensive and often times inefficient, under-utilizing the information encapsulated in the training text sequences. In this paper, we present SpacTor, a new training procedure consisting of (1) a hybrid objective combining span corruption (SC) and token replacement detection (RTD), and (2) a two-stage curriculum that optimizes the hybrid objective over the initial $τ$ iterations, then transitions to standard SC loss. We show empirically that the effectiveness of the hybrid objective is tied to the two-stage pre-training schedule, and provide extensive analysis on why this is the case. In our experiments with encoder-decoder architectures (T5) on a variety of NLP tasks, SpacTor-T5 yields the same downstream performance as standard SC pre-training, while enabling a 50% reduction in pre-training iterations and 40% reduction in total FLOPs. Alternatively, given the same amount of computing budget, we find that SpacTor results in significantly improved downstream benchmark performance. |
| title | SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2401.13160 |