MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912961148223488 |
|---|---|
| author | Xu, Tianyu Kim, Sieun Zheng, Qianhui Xu, Ruoyu Ravi, Tejasvi Kulkarni, Anuva Passarella-Ward, Katrina Zhu, Junyi Kowdle, Adarsh |
| author_facet | Xu, Tianyu Kim, Sieun Zheng, Qianhui Xu, Ruoyu Ravi, Tejasvi Kulkarni, Anuva Passarella-Ward, Katrina Zhu, Junyi Kowdle, Adarsh |
| contents | In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10465 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR Xu, Tianyu Kim, Sieun Zheng, Qianhui Xu, Ruoyu Ravi, Tejasvi Kulkarni, Anuva Passarella-Ward, Katrina Zhu, Junyi Kowdle, Adarsh Sound Computer Vision and Pattern Recognition Human-Computer Interaction H.5.1; H.5.2; H.5.5; I.2.7; I.2.10 In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences. |
| title | MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR |
| topic | Sound Computer Vision and Pattern Recognition Human-Computer Interaction H.5.1; H.5.2; H.5.5; I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2603.10465 |