Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909829788860416 |
|---|---|
| author | Marinoni, Christian Gramaccioni, Riccardo Fosco Grassucci, Eleonora Comminiello, Danilo |
| author_facet | Marinoni, Christian Gramaccioni, Riccardo Fosco Grassucci, Eleonora Comminiello, Danilo |
| contents | The generation of sounding videos has seen significant advancements with the advent of diffusion models. However, existing methods often lack the fine-grained control needed to generate viewpoint-specific content from larger, immersive 360-degree environments. This limitation restricts the creation of audio-visual experiences that are aware of off-camera events. To the best of our knowledge, this is the first work to introduce a framework for controllable audio-visual generation, addressing this unexplored gap. Specifically, we propose a diffusion model by introducing a set of powerful conditioning signals derived from the full 360-degree space: a panoramic saliency map to identify regions of interest, a bounding-box-aware signed distance map to define the target viewpoint, and a descriptive caption of the entire scene. By integrating these controls, our model generates spatially-aware viewpoint videos and audios that are coherently influenced by the broader, unseen environmental context, introducing a strong controllability that is essential for realistic and immersive audio-visual generation. We show audiovisual examples proving the effectiveness of our framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information Marinoni, Christian Gramaccioni, Riccardo Fosco Grassucci, Eleonora Comminiello, Danilo Multimedia Artificial Intelligence Computer Vision and Pattern Recognition The generation of sounding videos has seen significant advancements with the advent of diffusion models. However, existing methods often lack the fine-grained control needed to generate viewpoint-specific content from larger, immersive 360-degree environments. This limitation restricts the creation of audio-visual experiences that are aware of off-camera events. To the best of our knowledge, this is the first work to introduce a framework for controllable audio-visual generation, addressing this unexplored gap. Specifically, we propose a diffusion model by introducing a set of powerful conditioning signals derived from the full 360-degree space: a panoramic saliency map to identify regions of interest, a bounding-box-aware signed distance map to define the target viewpoint, and a descriptive caption of the entire scene. By integrating these controls, our model generates spatially-aware viewpoint videos and audios that are coherently influenced by the broader, unseen environmental context, introducing a strong controllability that is essential for realistic and immersive audio-visual generation. We show audiovisual examples proving the effectiveness of our framework. |
| title | Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information |
| topic | Multimedia Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.06060 |