Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915132995534848 |
|---|---|
| author | Lindenmaier, Gabriel Papay, Sean Padó, Sebastian |
| author_facet | Lindenmaier, Gabriel Papay, Sean Padó, Sebastian |
| contents | Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00617 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures Lindenmaier, Gabriel Papay, Sean Padó, Sebastian Computation and Language Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters. |
| title | Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.00617 |