IncDSI: Incrementally Updatable Document Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916360015052800 |
|---|---|
| author | Kishore, Varsha Wan, Chao Lovelace, Justin Artzi, Yoav Weinberger, Kilian Q. |
| author_facet | Kishore, Varsha Wan, Chao Lovelace, Justin Artzi, Yoav Weinberger, Kilian Q. |
| contents | Differentiable Search Index is a recently proposed paradigm for document retrieval, that encodes information about a corpus of documents within the parameters of a neural network and directly maps queries to corresponding documents. These models have achieved state-of-the-art performances for document retrieval across many benchmarks. These kinds of models have a significant limitation: it is not easy to add new documents after a model is trained. We propose IncDSI, a method to add documents in real time (about 20-50ms per document), without retraining the model on the entire dataset (or even parts thereof). Instead we formulate the addition of documents as a constrained optimization problem that makes minimal changes to the network parameters. Although orders of magnitude faster, our approach is competitive with re-training the model on the whole dataset and enables the development of document retrieval systems that can be updated with new information in real-time. Our code for IncDSI is available at https://github.com/varshakishore/IncDSI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_10323 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | IncDSI: Incrementally Updatable Document Retrieval Kishore, Varsha Wan, Chao Lovelace, Justin Artzi, Yoav Weinberger, Kilian Q. Information Retrieval Computation and Language Machine Learning Differentiable Search Index is a recently proposed paradigm for document retrieval, that encodes information about a corpus of documents within the parameters of a neural network and directly maps queries to corresponding documents. These models have achieved state-of-the-art performances for document retrieval across many benchmarks. These kinds of models have a significant limitation: it is not easy to add new documents after a model is trained. We propose IncDSI, a method to add documents in real time (about 20-50ms per document), without retraining the model on the entire dataset (or even parts thereof). Instead we formulate the addition of documents as a constrained optimization problem that makes minimal changes to the network parameters. Although orders of magnitude faster, our approach is competitive with re-training the model on the whole dataset and enables the development of document retrieval systems that can be updated with new information in real-time. Our code for IncDSI is available at https://github.com/varshakishore/IncDSI. |
| title | IncDSI: Incrementally Updatable Document Retrieval |
| topic | Information Retrieval Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2307.10323 |