Tricks and Plug-ins for Gradient Boosting with Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909881455345664 |
|---|---|
| author | Fang, Biyi Vo, Truong Utke, Jean Klabjan, Diego |
| author_facet | Fang, Biyi Vo, Truong Utke, Jean Klabjan, Diego |
| contents | Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02924 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tricks and Plug-ins for Gradient Boosting with Transformers Fang, Biyi Vo, Truong Utke, Jean Klabjan, Diego Machine Learning 68T07, 68Q32 I.2.6; I.5.1; F.1.1 Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead. |
| title | Tricks and Plug-ins for Gradient Boosting with Transformers |
| topic | Machine Learning 68T07, 68Q32 I.2.6; I.5.1; F.1.1 |
| url | https://arxiv.org/abs/2508.02924 |