A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917638452543488 |
|---|---|
| author | Ozdel, Suleyman Rong, Yao Albaba, Berat Mert Kuo, Yen-Ling Wang, Xi Kasneci, Enkelejda |
| author_facet | Ozdel, Suleyman Rong, Yao Albaba, Berat Mert Kuo, Yen-Ling Wang, Xi Kasneci, Enkelejda |
| contents | Eye-tracking applications that utilize the human gaze in video understanding tasks have become increasingly important. To effectively automate the process of video analysis based on eye-tracking data, it is important to accurately replicate human gaze behavior. However, this task presents significant challenges due to the inherent complexity and ambiguity of human gaze patterns. In this work, we introduce a novel method for simulating human gaze behavior. Our approach uses a transformer-based reinforcement learning algorithm to train an agent that acts as a human observer, with the primary role of watching videos and simulating human gaze behavior. We employed an eye-tracking dataset gathered from videos generated by the VirtualHome simulator, with a primary focus on activity recognition. Our experimental results demonstrate the effectiveness of our gaze prediction method by highlighting its capability to replicate human gaze behavior and its applicability for downstream tasks where real human-gaze is used as input. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_07351 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos Ozdel, Suleyman Rong, Yao Albaba, Berat Mert Kuo, Yen-Ling Wang, Xi Kasneci, Enkelejda Computer Vision and Pattern Recognition Human-Computer Interaction Machine Learning Eye-tracking applications that utilize the human gaze in video understanding tasks have become increasingly important. To effectively automate the process of video analysis based on eye-tracking data, it is important to accurately replicate human gaze behavior. However, this task presents significant challenges due to the inherent complexity and ambiguity of human gaze patterns. In this work, we introduce a novel method for simulating human gaze behavior. Our approach uses a transformer-based reinforcement learning algorithm to train an agent that acts as a human observer, with the primary role of watching videos and simulating human gaze behavior. We employed an eye-tracking dataset gathered from videos generated by the VirtualHome simulator, with a primary focus on activity recognition. Our experimental results demonstrate the effectiveness of our gaze prediction method by highlighting its capability to replicate human gaze behavior and its applicability for downstream tasks where real human-gaze is used as input. |
| title | A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2404.07351 |