On Retrieval Augmentation and the Limitations of Language Model Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913294558691328 |
|---|---|
| author | Chiang, Ting-Rui Yu, Xinyan Velocity Robinson, Joshua Liu, Ollie Lee, Isabelle Yogatama, Dani |
| author_facet | Chiang, Ting-Rui Yu, Xinyan Velocity Robinson, Joshua Liu, Ollie Lee, Isabelle Yogatama, Dani |
| contents | Augmenting a language model (LM) with $k$-nearest neighbors ($k$NN) retrieval on its training data alone can decrease its perplexity, though the underlying reasons for this remain elusive. In this work, we rule out one previously posited possibility -- the "softmax bottleneck." We then create a new dataset to evaluate LM generalization ability in the setting where training data contains additional information that is not causally relevant. This task is challenging even for GPT-3.5 Turbo. We show that, for both GPT-2 and Mistral 7B, $k$NN retrieval augmentation consistently improves performance in this setting. Finally, to make $k$NN retrieval more accessible, we propose using a multi-layer perceptron model that maps datastore keys to values as a drop-in replacement for traditional retrieval. This reduces storage costs by over 25x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09615 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | On Retrieval Augmentation and the Limitations of Language Model Training Chiang, Ting-Rui Yu, Xinyan Velocity Robinson, Joshua Liu, Ollie Lee, Isabelle Yogatama, Dani Computation and Language Augmenting a language model (LM) with $k$-nearest neighbors ($k$NN) retrieval on its training data alone can decrease its perplexity, though the underlying reasons for this remain elusive. In this work, we rule out one previously posited possibility -- the "softmax bottleneck." We then create a new dataset to evaluate LM generalization ability in the setting where training data contains additional information that is not causally relevant. This task is challenging even for GPT-3.5 Turbo. We show that, for both GPT-2 and Mistral 7B, $k$NN retrieval augmentation consistently improves performance in this setting. Finally, to make $k$NN retrieval more accessible, we propose using a multi-layer perceptron model that maps datastore keys to values as a drop-in replacement for traditional retrieval. This reduces storage costs by over 25x. |
| title | On Retrieval Augmentation and the Limitations of Language Model Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09615 |