OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913865856450560 |
|---|---|
| author | Meng, Lingkai Shao, Yu Yuan, Long Lai, Longbin Cheng, Peng Yu, Wenyuan Zhang, Wenjie Lin, Xuemin Zhou, Jingren |
| author_facet | Meng, Lingkai Shao, Yu Yuan, Long Lai, Longbin Cheng, Peng Yu, Wenyuan Zhang, Wenjie Lin, Xuemin Zhou, Jingren |
| contents | Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23239 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software Meng, Lingkai Shao, Yu Yuan, Long Lai, Longbin Cheng, Peng Yu, Wenyuan Zhang, Wenjie Lin, Xuemin Zhou, Jingren Software Engineering Artificial Intelligence Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation. |
| title | OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23239 |