Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.07823 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917774666760192 |
|---|---|
| author | Svikhnushina, Ekaterina Pu, Pearl |
| author_facet | Svikhnushina, Ekaterina Pu, Pearl |
| contents | This paper explores the efficacy of online versus offline evaluation methods in assessing conversational chatbots, specifically comparing first-party direct interactions with third-party observational assessments. By extending a benchmarking dataset of user dialogs with empathetic chatbots with offline third-party evaluations, we present a systematic comparison between the feedback from online interactions and the more detached offline third-party evaluations. Our results reveal that offline human evaluations fail to capture the subtleties of human-chatbot interactions as effectively as online assessments. In comparison, automated third-party evaluations using a GPT-4 model offer a better approximation of first-party human judgments given detailed instructions. This study highlights the limitations of third-party evaluations in grasping the complexities of user experiences and advocates for the integration of direct interaction feedback in conversational AI evaluation to enhance system development and user satisfaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07823 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots Svikhnushina, Ekaterina Pu, Pearl Human-Computer Interaction Computation and Language This paper explores the efficacy of online versus offline evaluation methods in assessing conversational chatbots, specifically comparing first-party direct interactions with third-party observational assessments. By extending a benchmarking dataset of user dialogs with empathetic chatbots with offline third-party evaluations, we present a systematic comparison between the feedback from online interactions and the more detached offline third-party evaluations. Our results reveal that offline human evaluations fail to capture the subtleties of human-chatbot interactions as effectively as online assessments. In comparison, automated third-party evaluations using a GPT-4 model offer a better approximation of first-party human judgments given detailed instructions. This study highlights the limitations of third-party evaluations in grasping the complexities of user experiences and advocates for the integration of direct interaction feedback in conversational AI evaluation to enhance system development and user satisfaction. |
| title | Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots |
| topic | Human-Computer Interaction Computation and Language |
| url | https://arxiv.org/abs/2409.07823 |