Leveraging Language Model Capabilities for Sound Event Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929447518601216 |
|---|---|
| author | Wang, Hualei Mao, Jianguo Guo, Zhifang Wan, Jiarui Liu, Hong Wang, Xiangdong |
| author_facet | Wang, Hualei Mao, Jianguo Guo, Zhifang Wan, Jiarui Liu, Hong Wang, Xiangdong |
| contents | Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_11530 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Leveraging Language Model Capabilities for Sound Event Detection Wang, Hualei Mao, Jianguo Guo, Zhifang Wan, Jiarui Liu, Hong Wang, Xiangdong Sound Artificial Intelligence Audio and Speech Processing Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification. |
| title | Leveraging Language Model Capabilities for Sound Event Detection |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2308.11530 |