Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.05925 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913421097697280 |
|---|---|
| author | Afzal, Anum Kowsik, Alexander Fani, Rajna Matthes, Florian |
| author_facet | Afzal, Anum Kowsik, Alexander Fani, Rajna Matthes, Florian |
| contents | Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05925 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop Afzal, Anum Kowsik, Alexander Fani, Rajna Matthes, Florian Computation and Language Artificial Intelligence Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation. |
| title | Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05925 |