Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.21124 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917230628831232 |
|---|---|
| author | Dementyev, Artem Zulfikar, Wazeer Hersek, Sinan Getreuer, Pascal Kumar, Anurag Kumar, Vivek |
| author_facet | Dementyev, Artem Zulfikar, Wazeer Hersek, Sinan Getreuer, Pascal Kumar, Anurag Kumar, Vivek |
| contents | Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21124 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs Dementyev, Artem Zulfikar, Wazeer Hersek, Sinan Getreuer, Pascal Kumar, Anurag Kumar, Vivek Sound Artificial Intelligence Audio and Speech Processing Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array. |
| title | PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2601.21124 |