Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.15157 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917629302669312 |
|---|---|
| author | Zhang, Chaoyun Ma, Zicheng Wu, Yuhao He, Shilin Qin, Si Ma, Minghua Qin, Xiaoting Kang, Yu Liang, Yuyi Gou, Xiaoyu Xue, Yajie Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| author_facet | Zhang, Chaoyun Ma, Zicheng Wu, Yuhao He, Shilin Qin, Si Ma, Minghua Qin, Xiaoting Kang, Yu Liang, Yuyi Gou, Xiaoyu Xue, Yajie Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| contents | Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging large language models (LLMs). Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images.
We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an "ask me anything" experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_15157 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models Zhang, Chaoyun Ma, Zicheng Wu, Yuhao He, Shilin Qin, Si Ma, Minghua Qin, Xiaoting Kang, Yu Liang, Yuyi Gou, Xiaoyu Xue, Yajie Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi Software Engineering Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging large language models (LLMs). Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images. We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an "ask me anything" experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface. |
| title | AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2403.15157 |