Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.14862 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916922167132160 |
|---|---|
| author | Kopiczko, Dawid J. Blankevoort, Tijmen Asano, Yuki M. |
| author_facet | Kopiczko, Dawid J. Blankevoort, Tijmen Asano, Yuki M. |
| contents | Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose Bitune, a method that enhances pretrained decoder-only LLMs by incorporating bidirectional attention into prompt processing. We evaluate Bitune in instruction-tuning and question-answering settings, showing significant improvements in performance on commonsense reasoning, arithmetic, and language understanding tasks. Furthermore, extensive ablation studies validate the role of each component of the method, and demonstrate that Bitune is compatible with various parameter-efficient finetuning techniques and full model finetuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14862 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs Kopiczko, Dawid J. Blankevoort, Tijmen Asano, Yuki M. Computation and Language Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose Bitune, a method that enhances pretrained decoder-only LLMs by incorporating bidirectional attention into prompt processing. We evaluate Bitune in instruction-tuning and question-answering settings, showing significant improvements in performance on commonsense reasoning, arithmetic, and language understanding tasks. Furthermore, extensive ablation studies validate the role of each component of the method, and demonstrate that Bitune is compatible with various parameter-efficient finetuning techniques and full model finetuning. |
| title | Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.14862 |