Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917578270572544 |
|---|---|
| author | Roman, Adrian S. Balamurugan, Baladithya Pothuganti, Rithik |
| author_facet | Roman, Adrian S. Balamurugan, Baladithya Pothuganti, Rithik |
| contents | This technical report details our work towards building an enhanced audio-visual sound event localization and detection (SELD) network. We build on top of the audio-only SELDnet23 model and adapt it to be audio-visual by merging both audio and video information prior to the gated recurrent unit (GRU) of the audio-only network. Our model leverages YOLO and DETIC object detectors. We also build a framework that implements audio-visual data augmentation and audio-visual synthetic data generation. We deliver an audio-visual SELDnet system that outperforms the existing audio-visual SELD baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_17129 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes Roman, Adrian S. Balamurugan, Baladithya Pothuganti, Rithik Sound Artificial Intelligence Audio and Speech Processing This technical report details our work towards building an enhanced audio-visual sound event localization and detection (SELD) network. We build on top of the audio-only SELDnet23 model and adapt it to be audio-visual by merging both audio and video information prior to the gated recurrent unit (GRU) of the audio-only network. Our model leverages YOLO and DETIC object detectors. We also build a framework that implements audio-visual data augmentation and audio-visual synthetic data generation. We deliver an audio-visual SELDnet system that outperforms the existing audio-visual SELD baseline. |
| title | Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.17129 |