Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.01199 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909763691872256 |
|---|---|
| author | Yang, Xiaoran Du, Yuyang Chen, Kexin Liew, Soung Chang Lu, Jiamin Guo, Ziyu Liu, Xiaoyan Yang, Qun Xu, Shiqi Fan, Xingyu Pan, Yuchen Cui, Taoyong Deng, Hongyu Dudder, Boris Pan, Jianzhang Fang, Qun Heng, Pheng Ann |
| author_facet | Yang, Xiaoran Du, Yuyang Chen, Kexin Liew, Soung Chang Lu, Jiamin Guo, Ziyu Liu, Xiaoyan Yang, Qun Xu, Shiqi Fan, Xingyu Pan, Yuchen Cui, Taoyong Deng, Hongyu Dudder, Boris Pan, Jianzhang Fang, Qun Heng, Pheng Ann |
| contents | As Industry 4.0 progresses, flexible manufacturing has become a cornerstone of modern industrial systems, with equipment automation playing a pivotal role. However, existing control software for industrial equipment, typically reliant on graphical user interfaces (GUIs) that require human interactions such as mouse clicks or screen touches, poses significant barriers to the adoption of code-based equipment automation. Recently, Large Language Model-based General Computer Control (LLM-GCC) has emerged as a promising approach to automate GUI-based operations. However, industrial settings pose unique challenges, including visually diverse, domain-specific interfaces and mission-critical tasks demanding high precision. This paper introduces IndusGCC, the first dataset and benchmark tailored to LLM-GCC in industrial environments, encompassing 448 real-world tasks across seven domains, from robotic arm control to production line configuration. IndusGCC features multimodal human interaction data with the equipment software, providing robust supervision for GUI-level code generation. Additionally, we propose a novel evaluation framework with functional and structural metrics to assess LLM-generated control scripts. Experimental results on mainstream LLMs demonstrate both the potential of LLM-GCC and the challenges it faces, establishing a strong foundation for future research toward fully automated factories. Our data and code are publicly available at: \href{https://github.com/Golden-Arc/IndustrialLLM}{https://github.com/Golden-Arc/IndustrialLLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_01199 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation Yang, Xiaoran Du, Yuyang Chen, Kexin Liew, Soung Chang Lu, Jiamin Guo, Ziyu Liu, Xiaoyan Yang, Qun Xu, Shiqi Fan, Xingyu Pan, Yuchen Cui, Taoyong Deng, Hongyu Dudder, Boris Pan, Jianzhang Fang, Qun Heng, Pheng Ann Systems and Control As Industry 4.0 progresses, flexible manufacturing has become a cornerstone of modern industrial systems, with equipment automation playing a pivotal role. However, existing control software for industrial equipment, typically reliant on graphical user interfaces (GUIs) that require human interactions such as mouse clicks or screen touches, poses significant barriers to the adoption of code-based equipment automation. Recently, Large Language Model-based General Computer Control (LLM-GCC) has emerged as a promising approach to automate GUI-based operations. However, industrial settings pose unique challenges, including visually diverse, domain-specific interfaces and mission-critical tasks demanding high precision. This paper introduces IndusGCC, the first dataset and benchmark tailored to LLM-GCC in industrial environments, encompassing 448 real-world tasks across seven domains, from robotic arm control to production line configuration. IndusGCC features multimodal human interaction data with the equipment software, providing robust supervision for GUI-level code generation. Additionally, we propose a novel evaluation framework with functional and structural metrics to assess LLM-generated control scripts. Experimental results on mainstream LLMs demonstrate both the potential of LLM-GCC and the challenges it faces, establishing a strong foundation for future research toward fully automated factories. Our data and code are publicly available at: \href{https://github.com/Golden-Arc/IndustrialLLM}{https://github.com/Golden-Arc/IndustrialLLM. |
| title | IndusGCC: A Data Benchmark and Evaluation Framework for GUI-Based General Computer Control in Industrial Automation |
| topic | Systems and Control |
| url | https://arxiv.org/abs/2509.01199 |