FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917401057034240 |
|---|---|
| author | Jian, Xiangru Xu, Hao Pang, Wei Zhao, Xinjian Tao, Chengyu Zhang, Qixin Zhang, Xikun Zhang, Chao Deng, Guanzhi Xue, Alex Du, Juan Yu, Tianshu Tarr, Garth Song, Linqi Sun, Qiuzhuang Tao, Dacheng |
| author_facet | Jian, Xiangru Xu, Hao Pang, Wei Zhao, Xinjian Tao, Chengyu Zhang, Qixin Zhang, Xikun Zhang, Chao Deng, Guanzhi Xue, Alex Du, Juan Yu, Tianshu Tarr, Garth Song, Linqi Sun, Qiuzhuang Tao, Dacheng |
| contents | The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07413 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios Jian, Xiangru Xu, Hao Pang, Wei Zhao, Xinjian Tao, Chengyu Zhang, Qixin Zhang, Xikun Zhang, Chao Deng, Guanzhi Xue, Alex Du, Juan Yu, Tianshu Tarr, Garth Song, Linqi Sun, Qiuzhuang Tao, Dacheng Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web. |
| title | FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.07413 |