Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929651102777344 |
|---|---|
| author | Michalkiewicz, Mateusz Bai, Sheena Baktashmotlagh, Mahsa Jampani, Varun Balakrishnan, Guha |
| author_facet | Michalkiewicz, Mateusz Bai, Sheena Baktashmotlagh, Mahsa Jampani, Varun Balakrishnan, Guha |
| contents | In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19920 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models Michalkiewicz, Mateusz Bai, Sheena Baktashmotlagh, Mahsa Jampani, Varun Balakrishnan, Guha Computer Vision and Pattern Recognition 68T45 In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions. |
| title | Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models |
| topic | Computer Vision and Pattern Recognition 68T45 |
| url | https://arxiv.org/abs/2412.19920 |