Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914855147012096 |
|---|---|
| author | Ooi, Kenneth Watcharasupat, Karn N. Lam, Bhan Ong, Zhen-Ting Gan, Woon-Seng |
| author_facet | Ooi, Kenneth Watcharasupat, Karn N. Lam, Bhan Ong, Zhen-Ting Gan, Woon-Seng |
| contents | Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194\pm0.0012$ for the best-performing all-modality model, against $0.1217\pm0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_08342 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs Ooi, Kenneth Watcharasupat, Karn N. Lam, Bhan Ong, Zhen-Ting Gan, Woon-Seng Sound Audio and Speech Processing Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194\pm0.0012$ for the best-performing all-modality model, against $0.1217\pm0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability. |
| title | Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2303.08342 |