BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929334362570752 |
|---|---|
| author | Cao, Qingqing Min, Sewon Wang, Yizhong Hajishirzi, Hannaneh |
| author_facet | Cao, Qingqing Min, Sewon Wang, Yizhong Hajishirzi, Hannaneh |
| contents | Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_01329 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models Cao, Qingqing Min, Sewon Wang, Yizhong Hajishirzi, Hannaneh Computation and Language Artificial Intelligence Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance. |
| title | BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2310.01329 |