Reading Between the Lines: Scalable User Feedback via Implicit Sentiment in Developer Prompts
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911169517715456 |
|---|---|
| author | Nam, Daye Salawa, Malgorzata Chandra, Satish |
| author_facet | Nam, Daye Salawa, Malgorzata Chandra, Satish |
| contents | Evaluating developer satisfaction with conversational AI assistants at scale is critical but challenging. User studies provide rich insights, but are unscalable, while large-scale quantitative signals from logs or in-product ratings are often too shallow or sparse to be reliable. To address this gap, we propose and evaluate a new approach: using sentiment analysis of developer prompts to identify implicit signals of user satisfaction. With an analysis of industrial usage logs of 372 professional developers, we show that this approach can identify a signal in ~8% of all interactions, a rate more than 13 times higher than explicit user feedback, with reasonable accuracy even with an off-the-shelf sentiment analysis approach. This new practical approach to complement existing feedback channels would open up new directions for building a more comprehensive understanding of the developer experience at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_18361 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reading Between the Lines: Scalable User Feedback via Implicit Sentiment in Developer Prompts Nam, Daye Salawa, Malgorzata Chandra, Satish Software Engineering Artificial Intelligence Human-Computer Interaction Evaluating developer satisfaction with conversational AI assistants at scale is critical but challenging. User studies provide rich insights, but are unscalable, while large-scale quantitative signals from logs or in-product ratings are often too shallow or sparse to be reliable. To address this gap, we propose and evaluate a new approach: using sentiment analysis of developer prompts to identify implicit signals of user satisfaction. With an analysis of industrial usage logs of 372 professional developers, we show that this approach can identify a signal in ~8% of all interactions, a rate more than 13 times higher than explicit user feedback, with reasonable accuracy even with an off-the-shelf sentiment analysis approach. This new practical approach to complement existing feedback channels would open up new directions for building a more comprehensive understanding of the developer experience at scale. |
| title | Reading Between the Lines: Scalable User Feedback via Implicit Sentiment in Developer Prompts |
| topic | Software Engineering Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2509.18361 |