FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913031252869120 |
|---|---|
| author | Takahashi, Jun Moteki, Atsunori Uchida, Akiyoshi Masui, Shoichi Yang, Fan Uchino, Kanji Song, Yueqi Bisk, Yonatan Neubig, Graham Kusajima, Ikuo Watanabe, Yasuto Ishida, Hiroyuki Nakagawa, Koki Jiang, Shan |
| author_facet | Takahashi, Jun Moteki, Atsunori Uchida, Akiyoshi Masui, Shoichi Yang, Fan Uchino, Kanji Song, Yueqi Bisk, Yonatan Neubig, Graham Kusajima, Ikuo Watanabe, Yasuto Ishida, Hiroyuki Nakagawa, Koki Jiang, Shan |
| contents | This paper introduces FieldWorkArena, a benchmark for agentic AI targeting real-world field work. With the recent increase in demand for agentic AI, they are built to detect and document safety hazards, procedural violations, and other critical incidents across real-world manufacturing and retail environments. Whereas most agentic AI benchmarks focus on performance in simulated or digital environments, our work addresses the fundamental challenge of evaluating agents in the real-world. In this paper, we improve the evaluation function from previous methods to assess the performance of agentic AI in diverse real-world tasks. Our dataset comprises on-site captured images/videos in factories, warehouses and retails. Tasks were meticulously developed through interviews with site workers and managers. Evaluation results confirmed that performance evaluation considering the characteristics of Multimodal LLM (MLLM) such as GPT-4o is feasible. Furthermore, this study identifies both the effectiveness and limitations of the proposed new evaluation methodology. The complete dataset and evaluation program are publicly accessible on the website (https://en-documents.research.global.fujitsu.com/fieldworkarena/) |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19662 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks Takahashi, Jun Moteki, Atsunori Uchida, Akiyoshi Masui, Shoichi Yang, Fan Uchino, Kanji Song, Yueqi Bisk, Yonatan Neubig, Graham Kusajima, Ikuo Watanabe, Yasuto Ishida, Hiroyuki Nakagawa, Koki Jiang, Shan Artificial Intelligence Computer Vision and Pattern Recognition This paper introduces FieldWorkArena, a benchmark for agentic AI targeting real-world field work. With the recent increase in demand for agentic AI, they are built to detect and document safety hazards, procedural violations, and other critical incidents across real-world manufacturing and retail environments. Whereas most agentic AI benchmarks focus on performance in simulated or digital environments, our work addresses the fundamental challenge of evaluating agents in the real-world. In this paper, we improve the evaluation function from previous methods to assess the performance of agentic AI in diverse real-world tasks. Our dataset comprises on-site captured images/videos in factories, warehouses and retails. Tasks were meticulously developed through interviews with site workers and managers. Evaluation results confirmed that performance evaluation considering the characteristics of Multimodal LLM (MLLM) such as GPT-4o is feasible. Furthermore, this study identifies both the effectiveness and limitations of the proposed new evaluation methodology. The complete dataset and evaluation program are publicly accessible on the website (https://en-documents.research.global.fujitsu.com/fieldworkarena/) |
| title | FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.19662 |