Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912412329836544 |
|---|---|
| author | Ryan, Fiona Bati, Ajay Lee, Sangmin Bolya, Daniel Hoffman, Judy Rehg, James M. |
| author_facet | Ryan, Fiona Bati, Ajay Lee, Sangmin Bolya, Daniel Hoffman, Judy Rehg, James M. |
| contents | We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene. Predicting a person's gaze target requires reasoning both about the person's appearance and the contents of the scene. Prior works have developed increasingly complex, hand-crafted pipelines for gaze target estimation that carefully fuse features from separate scene encoders, head encoders, and auxiliary models for signals like depth and pose. Motivated by the success of general-purpose feature extractors on a variety of visual tasks, we propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder. We extract a single feature representation for the scene, and apply a person-specific positional prompt to decode gaze with a lightweight module. We demonstrate state-of-the-art performance across several gaze benchmarks and provide extensive analysis to validate our design choices. Our code is available at: http://github.com/fkryan/gazelle . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09586 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders Ryan, Fiona Bati, Ajay Lee, Sangmin Bolya, Daniel Hoffman, Judy Rehg, James M. Computer Vision and Pattern Recognition We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene. Predicting a person's gaze target requires reasoning both about the person's appearance and the contents of the scene. Prior works have developed increasingly complex, hand-crafted pipelines for gaze target estimation that carefully fuse features from separate scene encoders, head encoders, and auxiliary models for signals like depth and pose. Motivated by the success of general-purpose feature extractors on a variety of visual tasks, we propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder. We extract a single feature representation for the scene, and apply a person-specific positional prompt to decode gaze with a lightweight module. We demonstrate state-of-the-art performance across several gaze benchmarks and provide extensive analysis to validate our design choices. Our code is available at: http://github.com/fkryan/gazelle . |
| title | Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.09586 |