End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915853801357312 |
|---|---|
| author | Dang, Nhi Le, Tung Nguyen, Huy Tien |
| author_facet | Dang, Nhi Le, Tung Nguyen, Huy Tien |
| contents | Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q\&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10570 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering Dang, Nhi Le, Tung Nguyen, Huy Tien Computation and Language Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q\&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention. |
| title | End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.10570 |