Caption: Generating Informative Content Labels for Image Buttons Using Next-Screen Context
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913985857585152 |
|---|---|
| author | Zhong, Mingyuan Mallavarapu, Ajit Nie, Qing |
| author_facet | Zhong, Mingyuan Mallavarapu, Ajit Nie, Qing |
| contents | We present Caption, an LLM-powered content label generation tool for visual interactive elements on mobile devices. Content labels are essential for screen readers to provide announcements for image-based elements, but are often missing or uninformative due to developer neglect. Automated captioning systems attempt to address this, but are limited to on-screen context, often resulting in inaccurate or unspecific labels. To generate more accurate and descriptive labels, Caption collects next-screen context on interactive elements by navigating to the destination screen that appears after an interaction and incorporating information from both the origin and destination screens. Preliminary results show Caption generates more accurate labels than both human annotators and an LLM baseline. We expect Caption to empower developers by providing actionable accessibility suggestions and directly support on-demand repairs by screen reader users. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08731 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Caption: Generating Informative Content Labels for Image Buttons Using Next-Screen Context Zhong, Mingyuan Mallavarapu, Ajit Nie, Qing Human-Computer Interaction We present Caption, an LLM-powered content label generation tool for visual interactive elements on mobile devices. Content labels are essential for screen readers to provide announcements for image-based elements, but are often missing or uninformative due to developer neglect. Automated captioning systems attempt to address this, but are limited to on-screen context, often resulting in inaccurate or unspecific labels. To generate more accurate and descriptive labels, Caption collects next-screen context on interactive elements by navigating to the destination screen that appears after an interaction and incorporating information from both the origin and destination screens. Preliminary results show Caption generates more accurate labels than both human annotators and an LLM baseline. We expect Caption to empower developers by providing actionable accessibility suggestions and directly support on-demand repairs by screen reader users. |
| title | Caption: Generating Informative Content Labels for Image Buttons Using Next-Screen Context |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2508.08731 |