THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915323302641664 |
|---|---|
| author | Patel, Udita Mulkar, Rutu Roberts, Jay Senthilkumar, Cibi Chakravarthy Gandhi, Sujay Zheng, Xiaofei Nayyar, Naumaan Kalra, Parul Castrillo, Rafael |
| author_facet | Patel, Udita Mulkar, Rutu Roberts, Jay Senthilkumar, Cibi Chakravarthy Gandhi, Sujay Zheng, Xiaofei Nayyar, Naumaan Kalra, Parul Castrillo, Rafael |
| contents | We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11626 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering Patel, Udita Mulkar, Rutu Roberts, Jay Senthilkumar, Cibi Chakravarthy Gandhi, Sujay Zheng, Xiaofei Nayyar, Naumaan Kalra, Parul Castrillo, Rafael Computation and Language Artificial Intelligence We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications. |
| title | THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.11626 |