Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912074996645888 |
|---|---|
| author | Packowski, Sarah Halilovic, Inge Schlotfeldt, Jenifer Smith, Trish |
| author_facet | Packowski, Sarah Halilovic, Inge Schlotfeldt, Jenifer Smith, Trish |
| contents | Retrieval-augmented generation (RAG) is a popular technique for using large language models (LLMs) to build customer-support, question-answering solutions. In this paper, we share our team's practical experience building and maintaining enterprise-scale RAG solutions that answer users' questions about our software based on product documentation. Our experience has not always matched the most common patterns in the RAG literature. This paper focuses on solution strategies that are modular and model-agnostic. For example, our experience over the past few years - using different search methods and LLMs, and many knowledge base collections - has been that simple changes to the way we create knowledge base content can have a huge impact on our RAG solutions' success. In this paper, we also discuss how we monitor and evaluate results. Common RAG benchmark evaluation techniques have not been useful for evaluating responses to novel user questions, so we have found a flexible, "human in the lead" approach is required. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12812 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective Packowski, Sarah Halilovic, Inge Schlotfeldt, Jenifer Smith, Trish Information Retrieval Artificial Intelligence Retrieval-augmented generation (RAG) is a popular technique for using large language models (LLMs) to build customer-support, question-answering solutions. In this paper, we share our team's practical experience building and maintaining enterprise-scale RAG solutions that answer users' questions about our software based on product documentation. Our experience has not always matched the most common patterns in the RAG literature. This paper focuses on solution strategies that are modular and model-agnostic. For example, our experience over the past few years - using different search methods and LLMs, and many knowledge base collections - has been that simple changes to the way we create knowledge base content can have a huge impact on our RAG solutions' success. In this paper, we also discuss how we monitor and evaluate results. Common RAG benchmark evaluation techniques have not been useful for evaluating responses to novel user questions, so we have found a flexible, "human in the lead" approach is required. |
| title | Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2410.12812 |