Evaluating Multiview Object Consistency in Humans and Image Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916387766665216 |
|---|---|
| author | Bonnen, Tyler Fu, Stephanie Bai, Yutong O'Connell, Thomas Friedman, Yoni Kanwisher, Nancy Tenenbaum, Joshua B. Efros, Alexei A. |
| author_facet | Bonnen, Tyler Fu, Stephanie Bai, Yutong O'Connell, Thomas Friedman, Yoni Kanwisher, Nancy Tenenbaum, Joshua B. Efros, Alexei A. |
| contents | We introduce a benchmark to directly evaluate the alignment between human observers and vision models on a 3D shape inference task. We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape: given a set of images, participants identify which contain the same/different objects, despite considerable viewpoint variation. We draw from a diverse range of images that include common objects (e.g., chairs) as well as abstract shapes (i.e., procedurally generated `nonsense' objects). After constructing over 2000 unique image sets, we administer these tasks to human participants, collecting 35K trials of behavioral data from over 500 participants. This includes explicit choice behaviors as well as intermediate measures, such as reaction time and gaze data. We then evaluate the performance of common vision models (e.g., DINOv2, MAE, CLIP). We find that humans outperform all models by a wide margin. Using a multi-scale evaluation approach, we identify underlying similarities and differences between models and humans: while human-model performance is correlated, humans allocate more time/processing on challenging trials. All images, data, and code can be accessed via our project page. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05862 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Multiview Object Consistency in Humans and Image Models Bonnen, Tyler Fu, Stephanie Bai, Yutong O'Connell, Thomas Friedman, Yoni Kanwisher, Nancy Tenenbaum, Joshua B. Efros, Alexei A. Computer Vision and Pattern Recognition We introduce a benchmark to directly evaluate the alignment between human observers and vision models on a 3D shape inference task. We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape: given a set of images, participants identify which contain the same/different objects, despite considerable viewpoint variation. We draw from a diverse range of images that include common objects (e.g., chairs) as well as abstract shapes (i.e., procedurally generated `nonsense' objects). After constructing over 2000 unique image sets, we administer these tasks to human participants, collecting 35K trials of behavioral data from over 500 participants. This includes explicit choice behaviors as well as intermediate measures, such as reaction time and gaze data. We then evaluate the performance of common vision models (e.g., DINOv2, MAE, CLIP). We find that humans outperform all models by a wide margin. Using a multi-scale evaluation approach, we identify underlying similarities and differences between models and humans: while human-model performance is correlated, humans allocate more time/processing on challenging trials. All images, data, and code can be accessed via our project page. |
| title | Evaluating Multiview Object Consistency in Humans and Image Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.05862 |