The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friends
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917073027858432 |
|---|---|
| author | Policzer, Nico Braunstein, Cameron Toneva, Mariya |
| author_facet | Policzer, Nico Braunstein, Cameron Toneva, Mariya |
| contents | Recent studies on audio models show brain-tuning - fine-tuning models to better predict corresponding fMRI activity - improves brain alignment and increases performance on downstream semantic and audio tasks. We extend this approach to a multimodal audio-video model to enhance social cognition, targeting the Superior Temporal Sulcus (STS), a key region for social processing, while subjects watch Friends. We find significant increases in brain alignment to the STS and an adjacent ROI, as well as improvements to a social cognition task related to the training data - sarcasm detection in sitcoms. In summary, our study extends brain-tuning to the multi-modal domain, demonstrating improvements to a downstream task after tuning to a relevant functional region. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07988 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friends Policzer, Nico Braunstein, Cameron Toneva, Mariya Artificial Intelligence Recent studies on audio models show brain-tuning - fine-tuning models to better predict corresponding fMRI activity - improves brain alignment and increases performance on downstream semantic and audio tasks. We extend this approach to a multimodal audio-video model to enhance social cognition, targeting the Superior Temporal Sulcus (STS), a key region for social processing, while subjects watch Friends. We find significant increases in brain alignment to the STS and an adjacent ROI, as well as improvements to a social cognition task related to the training data - sarcasm detection in sitcoms. In summary, our study extends brain-tuning to the multi-modal domain, demonstrating improvements to a downstream task after tuning to a relevant functional region. |
| title | The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friends |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.07988 |