AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910999646306304 |
|---|---|
| author | Jiang, Xilin Dindar, Sukru Samet Choudhari, Vishal Bickel, Stephan Mehta, Ashesh McKhann, Guy M Friedman, Daniel Flinker, Adeen Mesgarani, Nima |
| author_facet | Jiang, Xilin Dindar, Sukru Samet Choudhari, Vishal Bickel, Stephan Mehta, Ashesh McKhann, Guy M Friedman, Daniel Flinker, Adeen Mesgarani, Nima |
| contents | Auditory foundation models, including auditory large language models (LLMs), process all sound inputs equally, independent of listener perception. However, human auditory perception is inherently selective: listeners focus on specific speakers while ignoring others in complex auditory scenes. Existing models do not incorporate this selectivity, limiting their ability to generate perception-aligned responses. To address this, we introduce Intention-Informed Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM (AAD-LLM), a prototype system that integrates brain signals to infer listener attention. AAD-LLM extends an auditory LLM by incorporating intracranial electroencephalography (iEEG) recordings to decode which speaker a listener is attending to and refine responses accordingly. The model first predicts the attended speaker from neural activity, then conditions response generation on this inferred attentional state. We evaluate AAD-LLM on speaker description, speech transcription and extraction, and question answering in multitalker scenarios, with both objective and subjective ratings showing improved alignment with listener intention. By taking a first step toward intention-aware auditory AI, this work explores a new paradigm where listener perception informs machine listening, paving the way for future listener-centered auditory systems. Demo and code available: https://aad-llm.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16794 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AAD-LLM: Neural Attention-Driven Auditory Scene Understanding Jiang, Xilin Dindar, Sukru Samet Choudhari, Vishal Bickel, Stephan Mehta, Ashesh McKhann, Guy M Friedman, Daniel Flinker, Adeen Mesgarani, Nima Sound Artificial Intelligence Computation and Language Human-Computer Interaction Audio and Speech Processing Auditory foundation models, including auditory large language models (LLMs), process all sound inputs equally, independent of listener perception. However, human auditory perception is inherently selective: listeners focus on specific speakers while ignoring others in complex auditory scenes. Existing models do not incorporate this selectivity, limiting their ability to generate perception-aligned responses. To address this, we introduce Intention-Informed Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM (AAD-LLM), a prototype system that integrates brain signals to infer listener attention. AAD-LLM extends an auditory LLM by incorporating intracranial electroencephalography (iEEG) recordings to decode which speaker a listener is attending to and refine responses accordingly. The model first predicts the attended speaker from neural activity, then conditions response generation on this inferred attentional state. We evaluate AAD-LLM on speaker description, speech transcription and extraction, and question answering in multitalker scenarios, with both objective and subjective ratings showing improved alignment with listener intention. By taking a first step toward intention-aware auditory AI, this work explores a new paradigm where listener perception informs machine listening, paving the way for future listener-centered auditory systems. Demo and code available: https://aad-llm.github.io. |
| title | AAD-LLM: Neural Attention-Driven Auditory Scene Understanding |
| topic | Sound Artificial Intelligence Computation and Language Human-Computer Interaction Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.16794 |