Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.09277 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909290182213632 |
|---|---|
| author | Chaudhary, Daksh Vadlamani, Sri Lakshmi Thomas, Dimple Nejati, Shiva Sabetzadeh, Mehrdad |
| author_facet | Chaudhary, Daksh Vadlamani, Sri Lakshmi Thomas, Dimple Nejati, Shiva Sabetzadeh, Mehrdad |
| contents | This paper presents our experience developing a Llama-based chatbot for question answering about continuous integration and continuous delivery (CI/CD) at Ericsson, a multinational telecommunications company. Our chatbot is designed to handle the specificities of CI/CD documents at Ericsson, employing a retrieval-augmented generation (RAG) model to enhance accuracy and relevance. Our empirical evaluation of the chatbot on industrial CI/CD-related questions indicates that an ensemble retriever, combining BM25 and embedding retrievers, yields the best performance. When evaluated against a ground truth of 72 CI/CD questions and answers at Ericsson, our most accurate chatbot configuration provides fully correct answers for 61.11% of the questions, partially correct answers for 26.39%, and incorrect answers for 12.50%. Through an error analysis of the partially correct and incorrect answers, we discuss the underlying causes of inaccuracies and provide insights for further refinement. We also reflect on lessons learned and suggest future directions for further improving our chatbot's accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09277 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson Chaudhary, Daksh Vadlamani, Sri Lakshmi Thomas, Dimple Nejati, Shiva Sabetzadeh, Mehrdad Software Engineering This paper presents our experience developing a Llama-based chatbot for question answering about continuous integration and continuous delivery (CI/CD) at Ericsson, a multinational telecommunications company. Our chatbot is designed to handle the specificities of CI/CD documents at Ericsson, employing a retrieval-augmented generation (RAG) model to enhance accuracy and relevance. Our empirical evaluation of the chatbot on industrial CI/CD-related questions indicates that an ensemble retriever, combining BM25 and embedding retrievers, yields the best performance. When evaluated against a ground truth of 72 CI/CD questions and answers at Ericsson, our most accurate chatbot configuration provides fully correct answers for 61.11% of the questions, partially correct answers for 26.39%, and incorrect answers for 12.50%. Through an error analysis of the partially correct and incorrect answers, we discuss the underlying causes of inaccuracies and provide insights for further refinement. We also reflect on lessons learned and suggest future directions for further improving our chatbot's accuracy. |
| title | Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2408.09277 |