Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2602.01390 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909018990051328 |
|---|---|
| author | Do, Lana Jung, Gio Barajas, Juvenal Francisco Scott, Andrew Taylor Ihorn, Shasta Blum, Alexander Mario Athitsos, Vassilis Yoon, Ilmi |
| author_facet | Do, Lana Jung, Gio Barajas, Juvenal Francisco Scott, Andrew Taylor Ihorn, Shasta Blum, Alexander Mario Athitsos, Vassilis Yoon, Ilmi |
| contents | Digital video is central to communication, education, and entertainment, but without audio description (AD), blind and low-vision users are excluded. While crowdsourced platforms and vision-language models (VLMs) expand AD production, quality is rarely checked systematically. Existing evaluations rely on NLP metrics and short-clip guidelines, leaving open the question of how to assess long-form AD quality at scale. To address this, we developed a methodological workflow using Item Response Theory to evaluate VLM and human rater proficiency against expert-established ground truth. Evaluations were based on a six-dimensional framework, grounded in professional guidelines and shaped by insights from our accessibility experts and blind consultants. Findings suggest that top-performing VLMs can approximate ground-truth ratings at levels comparable to human raters. However, qualitative analysis reveals that VLM reasoning is less reliable and actionable than that of human respondents. These insights underscore the potential of hybrid evaluation systems that leverage VLMs alongside human oversight, offering a path toward scalable AD quality control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01390 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters Do, Lana Jung, Gio Barajas, Juvenal Francisco Scott, Andrew Taylor Ihorn, Shasta Blum, Alexander Mario Athitsos, Vassilis Yoon, Ilmi Human-Computer Interaction Artificial Intelligence Digital video is central to communication, education, and entertainment, but without audio description (AD), blind and low-vision users are excluded. While crowdsourced platforms and vision-language models (VLMs) expand AD production, quality is rarely checked systematically. Existing evaluations rely on NLP metrics and short-clip guidelines, leaving open the question of how to assess long-form AD quality at scale. To address this, we developed a methodological workflow using Item Response Theory to evaluate VLM and human rater proficiency against expert-established ground truth. Evaluations were based on a six-dimensional framework, grounded in professional guidelines and shaped by insights from our accessibility experts and blind consultants. Findings suggest that top-performing VLMs can approximate ground-truth ratings at levels comparable to human raters. However, qualitative analysis reveals that VLM reasoning is less reliable and actionable than that of human respondents. These insights underscore the potential of hybrid evaluation systems that leverage VLMs alongside human oversight, offering a path toward scalable AD quality control. |
| title | Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2602.01390 |