SeeingSounds: Learning Audio-to-Visual Alignment via Text
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909843354288128 |
|---|---|
| author | Carnemolla, Simone Pennisi, Matteo Russo, Chiara Palazzo, Simone Giordano, Daniela Spampinato, Concetto |
| author_facet | Carnemolla, Simone Pennisi, Matteo Russo, Chiara Palazzo, Simone Giordano, Daniela Spampinato, Concetto |
| contents | We introduce SeeingSounds, a lightweight and modular framework for audio-to-image generation that leverages the interplay between audio, language, and vision-without requiring any paired audio-visual data or training on visual generative models. Rather than treating audio as a substitute for text or relying solely on audio-to-text mappings, our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model. This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception. The model operates on frozen diffusion backbones and trains only lightweight adapters, enabling efficient and scalable learning. Moreover, it supports fine-grained and interpretable control through procedural text prompt generation, where audio transformations (e.g., volume or pitch shifts) translate into descriptive prompts (e.g., "a distant thunder") that guide visual outputs. Extensive experiments across standard benchmarks confirm that SeeingSounds outperforms existing methods in both zero-shot and supervised settings, establishing a new state of the art in controllable audio-to-visual generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11738 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SeeingSounds: Learning Audio-to-Visual Alignment via Text Carnemolla, Simone Pennisi, Matteo Russo, Chiara Palazzo, Simone Giordano, Daniela Spampinato, Concetto Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia We introduce SeeingSounds, a lightweight and modular framework for audio-to-image generation that leverages the interplay between audio, language, and vision-without requiring any paired audio-visual data or training on visual generative models. Rather than treating audio as a substitute for text or relying solely on audio-to-text mappings, our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model. This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception. The model operates on frozen diffusion backbones and trains only lightweight adapters, enabling efficient and scalable learning. Moreover, it supports fine-grained and interpretable control through procedural text prompt generation, where audio transformations (e.g., volume or pitch shifts) translate into descriptive prompts (e.g., "a distant thunder") that guide visual outputs. Extensive experiments across standard benchmarks confirm that SeeingSounds outperforms existing methods in both zero-shot and supervised settings, establishing a new state of the art in controllable audio-to-visual generation. |
| title | SeeingSounds: Learning Audio-to-Visual Alignment via Text |
| topic | Sound Artificial Intelligence Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2510.11738 |