Substance over Style: Evaluating Proactive Conversational Coaching Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908441450119168 |
|---|---|
| author | Srinivas, Vidya Xu, Xuhai Liu, Xin Ayush, Kumar Galatzer-Levy, Isaac Patel, Shwetak McDuff, Daniel Althoff, Tim |
| author_facet | Srinivas, Vidya Xu, Xuhai Liu, Xin Ayush, Kumar Galatzer-Levy, Isaac Patel, Shwetak McDuff, Daniel Althoff, Tim |
| contents | While NLP research has made strides in conversational tasks, many approaches focus on single-turn responses with well-defined objectives or evaluation criteria. In contrast, coaching presents unique challenges with initially undefined goals that evolve through multi-turn interactions, subjective evaluation criteria, mixed-initiative dialogue. In this work, we describe and implement five multi-turn coaching agents that exhibit distinct conversational styles, and evaluate them through a user study, collecting first-person feedback on 155 conversations. We find that users highly value core functionality, and that stylistic components in absence of core components are viewed negatively. By comparing user feedback with third-person evaluations from health experts and an LM, we reveal significant misalignment across evaluation approaches. Our findings provide insights into design and evaluation of conversational coaching agents and contribute toward improving human-centered NLP applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19328 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Substance over Style: Evaluating Proactive Conversational Coaching Agents Srinivas, Vidya Xu, Xuhai Liu, Xin Ayush, Kumar Galatzer-Levy, Isaac Patel, Shwetak McDuff, Daniel Althoff, Tim Computation and Language Artificial Intelligence While NLP research has made strides in conversational tasks, many approaches focus on single-turn responses with well-defined objectives or evaluation criteria. In contrast, coaching presents unique challenges with initially undefined goals that evolve through multi-turn interactions, subjective evaluation criteria, mixed-initiative dialogue. In this work, we describe and implement five multi-turn coaching agents that exhibit distinct conversational styles, and evaluate them through a user study, collecting first-person feedback on 155 conversations. We find that users highly value core functionality, and that stylistic components in absence of core components are viewed negatively. By comparing user feedback with third-person evaluations from health experts and an LM, we reveal significant misalignment across evaluation approaches. Our findings provide insights into design and evaluation of conversational coaching agents and contribute toward improving human-centered NLP applications. |
| title | Substance over Style: Evaluating Proactive Conversational Coaching Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.19328 |