Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911583878250496 |
|---|---|
| author | Choi, Jae Young Kim, Seon Gyeom Yoon, Hyungjun Lee, Taeckyung Lee, Donggun Chung, Jaeryung Kil, Jihyung Rossi, Ryan Lee, Sung-Ju Lee, Tak Yeon |
| author_facet | Choi, Jae Young Kim, Seon Gyeom Yoon, Hyungjun Lee, Taeckyung Lee, Donggun Chung, Jaeryung Kil, Jihyung Rossi, Ryan Lee, Sung-Ju Lee, Tak Yeon |
| contents | Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09585 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition Choi, Jae Young Kim, Seon Gyeom Yoon, Hyungjun Lee, Taeckyung Lee, Donggun Chung, Jaeryung Kil, Jihyung Rossi, Ryan Lee, Sung-Ju Lee, Tak Yeon Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts. |
| title | Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition |
| topic | Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.09585 |