OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909996922437632 |
|---|---|
| author | Popov, Maxim Kurkova, Regina Iumanov, Mikhail Mahmoud, Jaafar Kolyubin, Sergey |
| author_facet | Popov, Maxim Kurkova, Regina Iumanov, Mikhail Mahmoud, Jaafar Kolyubin, Sergey |
| contents | Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ, and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Project page is available at https://be2rlab.github.io/OSMa-Bench/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10331 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions Popov, Maxim Kurkova, Regina Iumanov, Mikhail Mahmoud, Jaafar Kolyubin, Sergey Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Robotics Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ, and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Project page is available at https://be2rlab.github.io/OSMa-Bench/. |
| title | OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Robotics |
| url | https://arxiv.org/abs/2503.10331 |