Probing the 3D Awareness of Visual Foundation Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911837683974144 |
|---|---|
| author | Banani, Mohamed El Raj, Amit Maninis, Kevis-Kokitsi Kar, Abhishek Li, Yuanzhen Rubinstein, Michael Sun, Deqing Guibas, Leonidas Johnson, Justin Jampani, Varun |
| author_facet | Banani, Mohamed El Raj, Amit Maninis, Kevis-Kokitsi Kar, Abhishek Li, Yuanzhen Rubinstein, Michael Sun, Deqing Guibas, Leonidas Johnson, Justin Jampani, Varun |
| contents | Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08636 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Probing the 3D Awareness of Visual Foundation Models Banani, Mohamed El Raj, Amit Maninis, Kevis-Kokitsi Kar, Abhishek Li, Yuanzhen Rubinstein, Michael Sun, Deqing Guibas, Leonidas Johnson, Justin Jampani, Varun Computer Vision and Pattern Recognition Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d. |
| title | Probing the 3D Awareness of Visual Foundation Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.08636 |