Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912623824470016 |
|---|---|
| author | Yang, Mengyu Chen, Yiming Pei, Haozheng Agarwal, Siddhant Vasudevan, Arun Balajee Hays, James |
| author_facet | Yang, Mengyu Chen, Yiming Pei, Haozheng Agarwal, Siddhant Vasudevan, Arun Balajee Hays, James |
| contents | Can a model distinguish between the sound of a spoon hitting a hardwood floor versus a carpeted one? Everyday object interactions produce sounds unique to the objects involved. We introduce the sounding object detection task to evaluate a model's ability to link these sounds to the objects directly involved. Inspired by human perception, our multimodal object-aware framework learns from in-the-wild egocentric videos. To encourage an object-centric approach, we first develop an automatic pipeline to compute segmentation masks of the objects involved to guide the model's focus during training towards the most informative regions of the interaction. A slot attention visual encoder is used to further enforce an object prior. We demonstrate state of the art performance on our new task along with existing multimodal action understanding tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_02313 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions Yang, Mengyu Chen, Yiming Pei, Haozheng Agarwal, Siddhant Vasudevan, Arun Balajee Hays, James Computer Vision and Pattern Recognition Can a model distinguish between the sound of a spoon hitting a hardwood floor versus a carpeted one? Everyday object interactions produce sounds unique to the objects involved. We introduce the sounding object detection task to evaluate a model's ability to link these sounds to the objects directly involved. Inspired by human perception, our multimodal object-aware framework learns from in-the-wild egocentric videos. To encourage an object-centric approach, we first develop an automatic pipeline to compute segmentation masks of the objects involved to guide the model's focus during training towards the most informative regions of the interaction. A slot attention visual encoder is used to further enforce an object prior. We demonstrate state of the art performance on our new task along with existing multimodal action understanding tasks. |
| title | Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.02313 |