Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911986964496384 |
|---|---|
| author | He, Junqing Pan, Kunhao Dong, Xiaoqun Song, Zhuoyang Liu, Yibo Sun, Qianguo Liang, Yuxin Wang, Hao Zhang, Enming Zhang, Jiaxing |
| author_facet | He, Junqing Pan, Kunhao Dong, Xiaoqun Song, Zhuoyang Liu, Yibo Sun, Qianguo Liang, Yuxin Wang, Hao Zhang, Enming Zhang, Jiaxing |
| contents | While large language models (LLMs) are equipped with longer text input capabilities than before, they are struggling to seek correct information in long contexts. The "lost in the middle" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA). Following these tasks, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. We release our model, Ziya-Reader to promote related research in the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09198 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training He, Junqing Pan, Kunhao Dong, Xiaoqun Song, Zhuoyang Liu, Yibo Sun, Qianguo Liang, Yuxin Wang, Hao Zhang, Enming Zhang, Jiaxing Computation and Language Artificial Intelligence While large language models (LLMs) are equipped with longer text input capabilities than before, they are struggling to seek correct information in long contexts. The "lost in the middle" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA). Following these tasks, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. We release our model, Ziya-Reader to promote related research in the community. |
| title | Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2311.09198 |