A Transformer with Stack Attention
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913348802576384 |
|---|---|
| author | Li, Jiaoda White, Jennifer C. Sachan, Mrinmaya Cotterell, Ryan |
| author_facet | Li, Jiaoda White, Jennifer C. Sachan, Mrinmaya Cotterell, Ryan |
| contents | Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04515 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Transformer with Stack Attention Li, Jiaoda White, Jennifer C. Sachan, Mrinmaya Cotterell, Ryan Computation and Language Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages. |
| title | A Transformer with Stack Attention |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.04515 |