How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909202865192960 |
|---|---|
| author | Chavez, Mariana Arroyo Feanny, Molly Seita, Matthew Thompson, Bernard Delk, Keith Officer, Skyler Glasser, Abraham Kushalnagar, Raja Vogler, Christian |
| author_facet | Chavez, Mariana Arroyo Feanny, Molly Seita, Matthew Thompson, Bernard Delk, Keith Officer, Skyler Glasser, Abraham Kushalnagar, Raja Vogler, Christian |
| contents | This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge Chavez, Mariana Arroyo Feanny, Molly Seita, Matthew Thompson, Bernard Delk, Keith Officer, Skyler Glasser, Abraham Kushalnagar, Raja Vogler, Christian Human-Computer Interaction This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers. |
| title | How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.10153 |