Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2502.15678 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912403182059520 |
|---|---|
| author | Buschoff, Luca M. Schulze Voudouris, Konstantinos Akata, Elif Bethge, Matthias Tenenbaum, Joshua B. Schulz, Eric |
| author_facet | Buschoff, Luca M. Schulze Voudouris, Konstantinos Akata, Elif Bethge, Matthias Tenenbaum, Joshua B. Schulz, Eric |
| contents | Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15678 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models Buschoff, Luca M. Schulze Voudouris, Konstantinos Akata, Elif Bethge, Matthias Tenenbaum, Joshua B. Schulz, Eric Machine Learning Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains. |
| title | Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.15678 |