Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909267928285184 |
|---|---|
| author | Chen, Changan Peng, Puyuan Baid, Ami Xue, Zihui Hsu, Wei-Ning Harwath, David Grauman, Kristen |
| author_facet | Chen, Changan Peng, Puyuan Baid, Ami Xue, Zihui Hsu, Wei-Ning Harwath, David Grauman, Kristen |
| contents | Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets, Ego4D and EPIC-KITCHENS, and we introduce Ego4D-Sounds -- 1.2M curated clips with action-audio correspondence. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our approach is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos Chen, Changan Peng, Puyuan Baid, Ami Xue, Zihui Hsu, Wei-Ning Harwath, David Grauman, Kristen Computer Vision and Pattern Recognition Artificial Intelligence Sound Audio and Speech Processing Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets, Ego4D and EPIC-KITCHENS, and we introduce Ego4D-Sounds -- 1.2M curated clips with action-audio correspondence. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our approach is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds. |
| title | Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.09272 |