HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913820426895360 |
|---|---|
| author | Khan, Muhammad Haris Cabrera, Miguel Altamirano Iarchuk, Dmitrii Mahmoud, Yara Trinitatova, Daria Tokmurziyev, Issatay Tsetserukou, Dzmitry |
| author_facet | Khan, Muhammad Haris Cabrera, Miguel Altamirano Iarchuk, Dmitrii Mahmoud, Yara Trinitatova, Daria Tokmurziyev, Issatay Tsetserukou, Dzmitry |
| contents | This paper introduces HapticVLM, a novel multimodal system that integrates vision-language reasoning with deep convolutional networks to enable real-time haptic feedback. HapticVLM leverages a ConvNeXt-based material recognition module to generate robust visual embeddings for accurate identification of object materials, while a state-of-the-art Vision-Language Model (Qwen2-VL-2B-Instruct) infers ambient temperature from environmental cues. The system synthesizes tactile sensations by delivering vibrotactile feedback through speakers and thermal cues via a Peltier module, thereby bridging the gap between visual perception and tactile experience. Experimental evaluations demonstrate an average recognition accuracy of 84.67% across five distinct auditory-tactile patterns and a temperature estimation accuracy of 86.7% based on a tolerance-based evaluation method with an 8°C margin of error across 15 scenarios. Although promising, the current study is limited by the use of a small set of prominent patterns and a modest participant pool. Future work will focus on expanding the range of tactile patterns and increasing user studies to further refine and validate the system's performance. Overall, HapticVLM presents a significant step toward context-aware, multimodal haptic interaction with potential applications in virtual reality, and assistive technologies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02569 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction Khan, Muhammad Haris Cabrera, Miguel Altamirano Iarchuk, Dmitrii Mahmoud, Yara Trinitatova, Daria Tokmurziyev, Issatay Tsetserukou, Dzmitry Robotics Human-Computer Interaction This paper introduces HapticVLM, a novel multimodal system that integrates vision-language reasoning with deep convolutional networks to enable real-time haptic feedback. HapticVLM leverages a ConvNeXt-based material recognition module to generate robust visual embeddings for accurate identification of object materials, while a state-of-the-art Vision-Language Model (Qwen2-VL-2B-Instruct) infers ambient temperature from environmental cues. The system synthesizes tactile sensations by delivering vibrotactile feedback through speakers and thermal cues via a Peltier module, thereby bridging the gap between visual perception and tactile experience. Experimental evaluations demonstrate an average recognition accuracy of 84.67% across five distinct auditory-tactile patterns and a temperature estimation accuracy of 86.7% based on a tolerance-based evaluation method with an 8°C margin of error across 15 scenarios. Although promising, the current study is limited by the use of a small set of prominent patterns and a modest participant pool. Future work will focus on expanding the range of tactile patterns and increasing user studies to further refine and validate the system's performance. Overall, HapticVLM presents a significant step toward context-aware, multimodal haptic interaction with potential applications in virtual reality, and assistive technologies. |
| title | HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction |
| topic | Robotics Human-Computer Interaction |
| url | https://arxiv.org/abs/2505.02569 |