Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909357652836352 |
|---|---|
| author | Khan, Ayman Asad Hasan, Md Toufique Kemell, Kai Kristian Rasku, Jussi Abrahamsson, Pekka |
| author_facet | Khan, Ayman Asad Hasan, Md Toufique Kemell, Kai Kristian Rasku, Jussi Abrahamsson, Pekka |
| contents | This paper presents an experience report on the development of Retrieval Augmented Generation (RAG) systems using PDF documents as the primary data source. The RAG architecture combines generative capabilities of Large Language Models (LLMs) with the precision of information retrieval. This approach has the potential to redefine how we interact with and augment both structured and unstructured knowledge in generative models to enhance transparency, accuracy, and contextuality of responses. The paper details the end-to-end pipeline, from data collection, preprocessing, to retrieval indexing and response generation, highlighting technical challenges and practical solutions. We aim to offer insights to researchers and practitioners developing similar systems using two distinct approaches: OpenAI's Assistant API with GPT Series and Llama's open-source models. The practical implications of this research lie in enhancing the reliability of generative AI systems in various sectors where domain-specific knowledge and real-time information retrieval is important. The Python code used in this work is also available at: https://github.com/GPT-Laboratory/RAG-LLM-Development-Guidebook-from-PDFs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15944 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report Khan, Ayman Asad Hasan, Md Toufique Kemell, Kai Kristian Rasku, Jussi Abrahamsson, Pekka Software Engineering Artificial Intelligence Information Retrieval This paper presents an experience report on the development of Retrieval Augmented Generation (RAG) systems using PDF documents as the primary data source. The RAG architecture combines generative capabilities of Large Language Models (LLMs) with the precision of information retrieval. This approach has the potential to redefine how we interact with and augment both structured and unstructured knowledge in generative models to enhance transparency, accuracy, and contextuality of responses. The paper details the end-to-end pipeline, from data collection, preprocessing, to retrieval indexing and response generation, highlighting technical challenges and practical solutions. We aim to offer insights to researchers and practitioners developing similar systems using two distinct approaches: OpenAI's Assistant API with GPT Series and Llama's open-source models. The practical implications of this research lie in enhancing the reliability of generative AI systems in various sectors where domain-specific knowledge and real-time information retrieval is important. The Python code used in this work is also available at: https://github.com/GPT-Laboratory/RAG-LLM-Development-Guidebook-from-PDFs. |
| title | Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report |
| topic | Software Engineering Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2410.15944 |