Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916995852664832 |
|---|---|
| author | An, Subin Ji, Yugyeong Kim, Junyoung Kook, Heejin Lu, Yang Seltzer, Josh |
| author_facet | An, Subin Ji, Yugyeong Kim, Junyoung Kook, Heejin Lu, Yang Seltzer, Josh |
| contents | Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06242 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses An, Subin Ji, Yugyeong Kim, Junyoung Kook, Heejin Lu, Yang Seltzer, Josh Computation and Language Artificial Intelligence Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment. |
| title | Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06242 |