In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913547752046592 |
|---|---|
| author | Lai, Bolin Liu, Miao Ryan, Fiona Rehg, James M. |
| author_facet | Lai, Bolin Liu, Miao Ryan, Fiona Rehg, James M. |
| contents | In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets - EGTEA Gaze+ and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds previous state-of-the-arts by a large margin. We also provide additional visualizations to support our claim that global-local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website (https://bolinlai.github.io/GLC-EgoGazeEst). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2208_04464 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation Lai, Bolin Liu, Miao Ryan, Fiona Rehg, James M. Computer Vision and Pattern Recognition In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets - EGTEA Gaze+ and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds previous state-of-the-arts by a large margin. We also provide additional visualizations to support our claim that global-local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website (https://bolinlai.github.io/GLC-EgoGazeEst). |
| title | In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2208.04464 |