Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.17771 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913879672487936 |
|---|---|
| author | Opper, Mattia Siddharth, N. |
| author_facet | Opper, Mattia Siddharth, N. |
| contents | We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17771 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Banyan: Improved Representation Learning with Explicit Structure Opper, Mattia Siddharth, N. Computation and Language We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments. |
| title | Banyan: Improved Representation Learning with Explicit Structure |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.17771 |