Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915510301491200 |
|---|---|
| author | Hasan, Md Toufique Waseem, Muhammad Kemell, Kai-Kristian Khan, Ayman Asad Saari, Mika Abrahamsson, Pekka |
| author_facet | Hasan, Md Toufique Waseem, Muhammad Kemell, Kai-Kristian Khan, Ayman Asad Saari, Mika Abrahamsson, Pekka |
| contents | Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20869 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation Hasan, Md Toufique Waseem, Muhammad Kemell, Kai-Kristian Khan, Ayman Asad Saari, Mika Abrahamsson, Pekka Software Engineering Artificial Intelligence Information Retrieval D.2.11; I.2.6; H.3.3 Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice. |
| title | Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation |
| topic | Software Engineering Artificial Intelligence Information Retrieval D.2.11; I.2.6; H.3.3 |
| url | https://arxiv.org/abs/2506.20869 |