Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909783203774464 |
|---|---|
| author | Elmansoury, Sary Mesabah, Islam Großmann, Gerrit Neigel, Peter Bhalwankar, Raj Kondermann, Daniel Vollmer, Sebastian J. |
| author_facet | Elmansoury, Sary Mesabah, Islam Großmann, Gerrit Neigel, Peter Bhalwankar, Raj Kondermann, Daniel Vollmer, Sebastian J. |
| contents | Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09732 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs Elmansoury, Sary Mesabah, Islam Großmann, Gerrit Neigel, Peter Bhalwankar, Raj Kondermann, Daniel Vollmer, Sebastian J. Computer Vision and Pattern Recognition Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems. |
| title | Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.09732 |