SAKR: Enhancing Retrieval-Augmented Generation via Streaming Algorithm and K-Means Clustering
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915287443439616 |
|---|---|
| author | Kang, Haoyu Zhu, Yuzhou Zhong, Yukun Wang, Ke |
| author_facet | Kang, Haoyu Zhu, Yuzhou Zhong, Yukun Wang, Ke |
| contents | Retrieval-augmented generation (RAG) has achieved significant success in information retrieval to assist large language models LLMs because it builds an external knowledge database. However, it also has many problems, it consumes a lot of memory because of the enormous database, and it cannot update the established index database in time when confronted with massive streaming data. To reduce the memory required for building the database and maintain accuracy simultaneously, we proposed a new approach integrating a streaming algorithm with k-means clustering into RAG. Our approach applied a streaming algorithm to update the index dynamically and reduce memory consumption. Additionally, the k-means algorithm clusters highly similar documents, and the query time would be shortened. We conducted comparative experiments on four methods, and the results indicated that RAG with streaming algorithm and k-means clusters outperforms traditional RAG in accuracy and memory, particularly when dealing with large-scale streaming data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_21300 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SAKR: Enhancing Retrieval-Augmented Generation via Streaming Algorithm and K-Means Clustering Kang, Haoyu Zhu, Yuzhou Zhong, Yukun Wang, Ke Information Retrieval Artificial Intelligence Retrieval-augmented generation (RAG) has achieved significant success in information retrieval to assist large language models LLMs because it builds an external knowledge database. However, it also has many problems, it consumes a lot of memory because of the enormous database, and it cannot update the established index database in time when confronted with massive streaming data. To reduce the memory required for building the database and maintain accuracy simultaneously, we proposed a new approach integrating a streaming algorithm with k-means clustering into RAG. Our approach applied a streaming algorithm to update the index dynamically and reduce memory consumption. Additionally, the k-means algorithm clusters highly similar documents, and the query time would be shortened. We conducted comparative experiments on four methods, and the results indicated that RAG with streaming algorithm and k-means clusters outperforms traditional RAG in accuracy and memory, particularly when dealing with large-scale streaming data. |
| title | SAKR: Enhancing Retrieval-Augmented Generation via Streaming Algorithm and K-Means Clustering |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2407.21300 |