VIBE: Video-Input Brain Encoder for fMRI Response Modeling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908466274107392 |
|---|---|
| author | Schad, Daniel Carlström Dixit, Shrey Keck, Janis Studenyak, Viktor Shpilevoi, Aleksandr Bicanski, Andrej |
| author_facet | Schad, Daniel Carlström Dixit, Shrey Keck, Janis Studenyak, Viktor Shpilevoi, Aleksandr Bicanski, Andrej |
| contents | We present VIBE, a two-stage Transformer that fuses multi-modal video, audio, and text features to predict fMRI activity. Representations from open-source models (Qwen2.5, BEATs, Whisper, SlowFast, V-JEPA) are merged by a modality-fusion transformer and temporally decoded by a prediction transformer with rotary embeddings. Trained on 65 hours of movie data from the CNeuroMod dataset and ensembled across 20 seeds, VIBE attains mean parcel-wise Pearson correlations of 0.3225 on in-distribution Friends S07 and 0.2125 on six out-of-distribution films. An earlier iteration of the same architecture obtained 0.3198 and 0.2096, respectively, winning Phase-1 and placing second overall in the Algonauts 2025 Challenge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17958 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VIBE: Video-Input Brain Encoder for fMRI Response Modeling Schad, Daniel Carlström Dixit, Shrey Keck, Janis Studenyak, Viktor Shpilevoi, Aleksandr Bicanski, Andrej Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition We present VIBE, a two-stage Transformer that fuses multi-modal video, audio, and text features to predict fMRI activity. Representations from open-source models (Qwen2.5, BEATs, Whisper, SlowFast, V-JEPA) are merged by a modality-fusion transformer and temporally decoded by a prediction transformer with rotary embeddings. Trained on 65 hours of movie data from the CNeuroMod dataset and ensembled across 20 seeds, VIBE attains mean parcel-wise Pearson correlations of 0.3225 on in-distribution Friends S07 and 0.2125 on six out-of-distribution films. An earlier iteration of the same architecture obtained 0.3198 and 0.2096, respectively, winning Phase-1 and placing second overall in the Algonauts 2025 Challenge. |
| title | VIBE: Video-Input Brain Encoder for fMRI Response Modeling |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.17958 |