Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912975540977664 |
|---|---|
| author | Luo, Weiyao Zheng, Suncong Xia, Heming Wang, Weikang Lei, Yan Liu, Tianyu Chen, Shuang Sui, Zhifang |
| author_facet | Luo, Weiyao Zheng, Suncong Xia, Heming Wang, Weikang Lei, Yan Liu, Tianyu Chen, Shuang Sui, Zhifang |
| contents | Large language models (LLMs) have shown promising efficacy across various tasks, becoming powerful tools in numerous aspects of human life. However, Transformer-based LLMs suffer a performance degradation when modeling long-term contexts due to they discard some information to reduce computational overhead. In this work, we propose a simple yet effective method to enable LLMs to take a deep breath, encouraging them to summarize information contained within discrete text chunks. Specifically, we segment the text into multiple chunks and insert special token <SR> at the end of each chunk. We then modify the attention mask to integrate the chunk's information into the corresponding <SR> token. This facilitates LLMs to interpret information not only from historical individual tokens but also from the <SR> token, aggregating the chunk's semantic information. Experiments on language modeling and out-of-domain downstream tasks validate the superiority of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10985 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens Luo, Weiyao Zheng, Suncong Xia, Heming Wang, Weikang Lei, Yan Liu, Tianyu Chen, Shuang Sui, Zhifang Computation and Language Large language models (LLMs) have shown promising efficacy across various tasks, becoming powerful tools in numerous aspects of human life. However, Transformer-based LLMs suffer a performance degradation when modeling long-term contexts due to they discard some information to reduce computational overhead. In this work, we propose a simple yet effective method to enable LLMs to take a deep breath, encouraging them to summarize information contained within discrete text chunks. Specifically, we segment the text into multiple chunks and insert special token <SR> at the end of each chunk. We then modify the attention mask to integrate the chunk's information into the corresponding <SR> token. This facilitates LLMs to interpret information not only from historical individual tokens but also from the <SR> token, aggregating the chunk's semantic information. Experiments on language modeling and out-of-domain downstream tasks validate the superiority of our approach. |
| title | Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.10985 |