Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915181646315520 |
|---|---|
| author | Suganthan, Paul Moiseev, Fedor Yan, Le Wu, Junru Ni, Jianmo Han, Jay Zitouni, Imed Alfonseca, Enrique Wang, Xuanhui Dong, Zhe |
| author_facet | Suganthan, Paul Moiseev, Fedor Yan, Le Wu, Junru Ni, Jianmo Han, Jay Zitouni, Imed Alfonseca, Enrique Wang, Xuanhui Dong, Zhe |
| contents | Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language processing. Specifically, encoder-only models remain dominant in tasks like classification, regression, and ranking. This is primarily due to the inherent structure of decoder-based models, which limits their direct applicability to these tasks. In this paper, we introduce Gemma Encoder, adapting the powerful Gemma decoder model to an encoder architecture, thereby unlocking its potential for a wider range of non-generative applications. To optimize the adaptation from decoder to encoder, we systematically analyze various pooling strategies, attention mechanisms, and hyperparameters (e.g., dropout rate). Furthermore, we benchmark Gemma Encoder against established approaches on the GLUE benchmarks, and MS MARCO ranking benchmark, demonstrating its effectiveness and versatility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02656 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks Suganthan, Paul Moiseev, Fedor Yan, Le Wu, Junru Ni, Jianmo Han, Jay Zitouni, Imed Alfonseca, Enrique Wang, Xuanhui Dong, Zhe Computation and Language Machine Learning Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language processing. Specifically, encoder-only models remain dominant in tasks like classification, regression, and ranking. This is primarily due to the inherent structure of decoder-based models, which limits their direct applicability to these tasks. In this paper, we introduce Gemma Encoder, adapting the powerful Gemma decoder model to an encoder architecture, thereby unlocking its potential for a wider range of non-generative applications. To optimize the adaptation from decoder to encoder, we systematically analyze various pooling strategies, attention mechanisms, and hyperparameters (e.g., dropout rate). Furthermore, we benchmark Gemma Encoder against established approaches on the GLUE benchmarks, and MS MARCO ranking benchmark, demonstrating its effectiveness and versatility. |
| title | Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.02656 |