MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915164013461504 |
|---|---|
| author | Jiang, Xi Li, Jian Deng, Hanqiu Liu, Yong Gao, Bin-Bin Zhou, Yifeng Li, Jialin Wang, Chengjie Zheng, Feng |
| author_facet | Jiang, Xi Li, Jian Deng, Hanqiu Liu, Yong Gao, Bin-Bin Zhou, Yifeng Li, Jialin Wang, Chengjie Zheng, Feng |
| contents | In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09453 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection Jiang, Xi Li, Jian Deng, Hanqiu Liu, Yong Gao, Bin-Bin Zhou, Yifeng Li, Jialin Wang, Chengjie Zheng, Feng Artificial Intelligence Computer Vision and Pattern Recognition In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research. |
| title | MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.09453 |