Automating construction safety inspections using a multi-modal vision-language RAG framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915533772816384 |
|---|---|
| author | Wang, Chenxin Shamsabadi, Elyas Asadi Chen, Zhaohui Shen, Luming Fini, Alireza Ahmadian Fard Dias-da-Costa, Daniel |
| author_facet | Wang, Chenxin Shamsabadi, Elyas Asadi Chen, Zhaohui Shen, Luming Fini, Alireza Ahmadian Fard Dias-da-Costa, Daniel |
| contents | Conventional construction safety inspection methods are often inefficient as they require navigating through large volume of information. Recent advances in large vision-language models (LVLMs) provide opportunities to automate safety inspections through enhanced visual and linguistic understanding. However, existing applications face limitations including irrelevant or unspecific responses, restricted modal inputs and hallucinations. Utilisation of Large Language Models (LLMs) for this purpose is constrained by availability of training data and frequently lack real-time adaptability. This study introduces SiteShield, a multi-modal LVLM-based Retrieval-Augmented Generation (RAG) framework for automating construction safety inspection reports by integrating visual and audio inputs. Using real-world data, SiteShield outperformed unimodal LLMs without RAG with an F1 score of 0.82, hamming loss of 0.04, precision of 0.76, and recall of 0.96. The findings indicate that SiteShield offers a novel pathway to enhance information retrieval and efficiency in generating safety reports. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04145 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automating construction safety inspections using a multi-modal vision-language RAG framework Wang, Chenxin Shamsabadi, Elyas Asadi Chen, Zhaohui Shen, Luming Fini, Alireza Ahmadian Fard Dias-da-Costa, Daniel Computer Vision and Pattern Recognition Computation and Language Information Retrieval Conventional construction safety inspection methods are often inefficient as they require navigating through large volume of information. Recent advances in large vision-language models (LVLMs) provide opportunities to automate safety inspections through enhanced visual and linguistic understanding. However, existing applications face limitations including irrelevant or unspecific responses, restricted modal inputs and hallucinations. Utilisation of Large Language Models (LLMs) for this purpose is constrained by availability of training data and frequently lack real-time adaptability. This study introduces SiteShield, a multi-modal LVLM-based Retrieval-Augmented Generation (RAG) framework for automating construction safety inspection reports by integrating visual and audio inputs. Using real-world data, SiteShield outperformed unimodal LLMs without RAG with an F1 score of 0.82, hamming loss of 0.04, precision of 0.76, and recall of 0.96. The findings indicate that SiteShield offers a novel pathway to enhance information retrieval and efficiency in generating safety reports. |
| title | Automating construction safety inspections using a multi-modal vision-language RAG framework |
| topic | Computer Vision and Pattern Recognition Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2510.04145 |