Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909644778110976 |
|---|---|
| author | Liu, Mengqiao Wang, Tevin Cohen, Cassandra A. Li, Sarah Xiong, Chenyan |
| author_facet | Liu, Mengqiao Wang, Tevin Cohen, Cassandra A. Li, Sarah Xiong, Chenyan |
| contents | Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interact with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then be interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, e.g., the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our code and data are at https://github.com/cxcscmu/LLM-Interviewer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15226 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews Liu, Mengqiao Wang, Tevin Cohen, Cassandra A. Li, Sarah Xiong, Chenyan Computation and Language Artificial Intelligence Human-Computer Interaction Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interact with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then be interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, e.g., the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our code and data are at https://github.com/cxcscmu/LLM-Interviewer. |
| title | Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2502.15226 |