Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916843346722816 |
|---|---|
| author | Kushwaha, Saksham Singh Ma, Jianbo Thomas, Mark R. P. Tian, Yapeng Bruni, Avery |
| author_facet | Kushwaha, Saksham Singh Ma, Jianbo Thomas, Mark R. P. Tian, Yapeng Bruni, Avery |
| contents | Spatial audio is a crucial component in creating immersive experiences. Traditional simulation-based approaches to generate spatial audio rely on expertise, have limited scalability, and assume independence between semantic and spatial information. To address these issues, we explore end-to-end spatial audio generation. We introduce and formulate a new task of generating first-order Ambisonics (FOA) given a sound category and sound source spatial location. We propose Diff-SAGe, an end-to-end, flow-based diffusion-transformer model for this task. Diff-SAGe utilizes a complex spectrogram representation for FOA, preserving the phase information crucial for accurate spatial cues. Additionally, a multi-conditional encoder integrates the input conditions into a unified representation, guiding the generation of FOA waveforms from noise. Through extensive evaluations on two datasets, we demonstrate that our method consistently outperforms traditional simulation-based baselines across both objective and subjective metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11299 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models Kushwaha, Saksham Singh Ma, Jianbo Thomas, Mark R. P. Tian, Yapeng Bruni, Avery Sound Audio and Speech Processing Spatial audio is a crucial component in creating immersive experiences. Traditional simulation-based approaches to generate spatial audio rely on expertise, have limited scalability, and assume independence between semantic and spatial information. To address these issues, we explore end-to-end spatial audio generation. We introduce and formulate a new task of generating first-order Ambisonics (FOA) given a sound category and sound source spatial location. We propose Diff-SAGe, an end-to-end, flow-based diffusion-transformer model for this task. Diff-SAGe utilizes a complex spectrogram representation for FOA, preserving the phase information crucial for accurate spatial cues. Additionally, a multi-conditional encoder integrates the input conditions into a unified representation, guiding the generation of FOA waveforms from noise. Through extensive evaluations on two datasets, we demonstrate that our method consistently outperforms traditional simulation-based baselines across both objective and subjective metrics. |
| title | Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.11299 |