Vision-Language Models under Cultural and Inclusive Considerations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917715176849408 |
|---|---|
| author | Karamolegkou, Antonia Rust, Phillip Cao, Yong Cui, Ruixiang Søgaard, Anders Hershcovich, Daniel |
| author_facet | Karamolegkou, Antonia Rust, Phillip Cao, Yong Cui, Ruixiang Søgaard, Anders Hershcovich, Daniel |
| contents | Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06177 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vision-Language Models under Cultural and Inclusive Considerations Karamolegkou, Antonia Rust, Phillip Cao, Yong Cui, Ruixiang Søgaard, Anders Hershcovich, Daniel Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Computers and Society Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available. |
| title | Vision-Language Models under Cultural and Inclusive Considerations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2407.06177 |