Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912557409763328 |
|---|---|
| author | Freitas, Diogo Håvardstun, Brigt Ferri, Cèsar Garigliotti, Darío Telle, Jan Arne Hernández-Orallo, José |
| author_facet | Freitas, Diogo Håvardstun, Brigt Ferri, Cèsar Garigliotti, Darío Telle, Jan Arne Hernández-Orallo, José |
| contents | Large language models have become multimodal, and many of them are said to integrate their modalities using common representations. If this were true, a drawing of a car as an image, for instance, should map to a similar area in the latent space as a textual description of the strokes that form the drawing. To explore this in a black-box access regime to these models, we propose the use of machine teaching, a theory that studies the minimal set of examples a teacher needs to choose so that the learner captures the concept. In this paper, we evaluate the complexity of teaching vision-language models a subset of objects in the Quick, Draw! dataset using two presentations: raw images as bitmaps and trace coordinates in TikZ format. The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations. But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors, suggesting that the simplicity of concepts may be an inherent property that transcends modality representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10583 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models Freitas, Diogo Håvardstun, Brigt Ferri, Cèsar Garigliotti, Darío Telle, Jan Arne Hernández-Orallo, José Computer Vision and Pattern Recognition Computation and Language I.2.7; I.2.10 Large language models have become multimodal, and many of them are said to integrate their modalities using common representations. If this were true, a drawing of a car as an image, for instance, should map to a similar area in the latent space as a textual description of the strokes that form the drawing. To explore this in a black-box access regime to these models, we propose the use of machine teaching, a theory that studies the minimal set of examples a teacher needs to choose so that the learner captures the concept. In this paper, we evaluate the complexity of teaching vision-language models a subset of objects in the Quick, Draw! dataset using two presentations: raw images as bitmaps and trace coordinates in TikZ format. The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations. But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors, suggesting that the simplicity of concepts may be an inherent property that transcends modality representations. |
| title | Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2505.10583 |