Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.20858 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912251875688448 |
|---|---|
| author | Liu, Xiaochuan Cheng, Xin Sun, Yuchong Wu, Xiaoxue Song, Ruihua Sun, Hao Zhang, Denghao |
| author_facet | Liu, Xiaochuan Cheng, Xin Sun, Yuchong Wu, Xiaoxue Song, Ruihua Sun, Hao Zhang, Denghao |
| contents | Imitating how humans move their gaze in a visual scene is a vital research problem for both visual understanding and psychology, kindling crucial applications such as building alive virtual characters. Previous studies aim to predict gaze trajectories when humans are free-viewing an image, searching for required targets, or looking for clues to answer questions in an image. While these tasks focus on visual-centric scenarios, humans move their gaze also along with audio signal inputs in more common scenarios. To fill this gap, we introduce a new task that predicts human gaze trajectories in a visual scene with synchronized audio inputs and provide a new dataset containing 20k gaze points from 8 subjects. To effectively integrate audio information and simulate the dynamic process of human gaze motion, we propose a novel learning framework called EyEar (Eye moving while Ear listening) based on physics-informed dynamics, which considers three key factors to predict gazes: eye inherent motion tendency, vision salient attraction, and audio semantic attraction. We also propose a probability density score to overcome the high individual variability of gaze trajectories, thereby improving the stabilization of optimization and the reliability of the evaluation. Experimental results show that EyEar outperforms all the baselines in the context of all evaluation metrics, thanks to the proposed components in the learning model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics Liu, Xiaochuan Cheng, Xin Sun, Yuchong Wu, Xiaoxue Song, Ruihua Sun, Hao Zhang, Denghao Multimedia Imitating how humans move their gaze in a visual scene is a vital research problem for both visual understanding and psychology, kindling crucial applications such as building alive virtual characters. Previous studies aim to predict gaze trajectories when humans are free-viewing an image, searching for required targets, or looking for clues to answer questions in an image. While these tasks focus on visual-centric scenarios, humans move their gaze also along with audio signal inputs in more common scenarios. To fill this gap, we introduce a new task that predicts human gaze trajectories in a visual scene with synchronized audio inputs and provide a new dataset containing 20k gaze points from 8 subjects. To effectively integrate audio information and simulate the dynamic process of human gaze motion, we propose a novel learning framework called EyEar (Eye moving while Ear listening) based on physics-informed dynamics, which considers three key factors to predict gazes: eye inherent motion tendency, vision salient attraction, and audio semantic attraction. We also propose a probability density score to overcome the high individual variability of gaze trajectories, thereby improving the stabilization of optimization and the reliability of the evaluation. Experimental results show that EyEar outperforms all the baselines in the context of all evaluation metrics, thanks to the proposed components in the learning model. |
| title | EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics |
| topic | Multimedia |
| url | https://arxiv.org/abs/2502.20858 |