ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914812363014144 |
|---|---|
| author | Wu, Shangyu Xiong, Ying Cui, Yufei Liu, Xue Tang, Buzhou Kuo, Tei-Wei Xue, Chun Jason |
| author_facet | Wu, Shangyu Xiong, Ying Cui, Yufei Liu, Xue Tang, Buzhou Kuo, Tei-Wei Xue, Chun Jason |
| contents | Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02993 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion Wu, Shangyu Xiong, Ying Cui, Yufei Liu, Xue Tang, Buzhou Kuo, Tei-Wei Xue, Chun Jason Computation and Language Artificial Intelligence Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks. |
| title | ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.02993 |