Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915493173002240 |
|---|---|
| author | Xie, Junyu Han, Tengda Bain, Max Nagrani, Arsha Khandelwal, Eshika Varol, Gül Xie, Weidi Zisserman, Andrew |
| author_facet | Xie, Junyu Han, Tengda Bain, Max Nagrani, Arsha Khandelwal, Eshika Varol, Gül Xie, Weidi Zisserman, Andrew |
| contents | Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages "shots" as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure -- an action score -- specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01020 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation Xie, Junyu Han, Tengda Bain, Max Nagrani, Arsha Khandelwal, Eshika Varol, Gül Xie, Weidi Zisserman, Andrew Computer Vision and Pattern Recognition Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages "shots" as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure -- an action score -- specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection. |
| title | Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.01020 |