GPT or BERT: why not both?
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912171154210816 |
|---|---|
| author | Charpentier, Lucas Georges Gabriel Samuel, David |
| author_facet | Charpentier, Lucas Georges Gabriel Samuel, David |
| contents | We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_24159 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GPT or BERT: why not both? Charpentier, Lucas Georges Gabriel Samuel, David Computation and Language We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code. |
| title | GPT or BERT: why not both? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.24159 |