Towards Interpretable Visual Decoding with Attention to Brain Representations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917301136130048 |
|---|---|
| author | Feng, Pinyuan Adeli, Hossein Guo, Wenxuan Cheng, Fan Hwang, Ethan Kriegeskorte, Nikolaus |
| author_facet | Feng, Pinyuan Adeli, Hossein Guo, Wenxuan Cheng, Fan Hwang, Ethan Kriegeskorte, Nikolaus |
| contents | Recent work has demonstrated that complex visual stimuli can be decoded from human brain activity using deep generative models, offering new ways to probe how the brain represents real-world scenes. However, many existing approaches first map brain signals into intermediate image or text feature spaces before guiding the generative process, which obscures the contributions of different brain areas to the final reconstruction output. In this work, we propose NeuroAdapter, a visual decoding framework that directly conditions a latent diffusion model on brain representations, bypassing the need for intermediate feature spaces. Our method demonstrates competitive visual reconstruction quality on public fMRI datasets compared to prior work, while providing greater transparency into how brain signals drive visual reconstruction. To this end, we introduce an Image-Brain BI-directional interpretability framework (IBBI) that analyzes cross-attention patterns across diffusion denoising steps to reveal how different cortical areas influence the unfolding generative trajectory. Our work highlights the potential of end-to-end brain-to-image reconstruction and establishes a path for interpretable neural decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23566 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Interpretable Visual Decoding with Attention to Brain Representations Feng, Pinyuan Adeli, Hossein Guo, Wenxuan Cheng, Fan Hwang, Ethan Kriegeskorte, Nikolaus Computer Vision and Pattern Recognition 68T07 I.2.0; I.4.9 Recent work has demonstrated that complex visual stimuli can be decoded from human brain activity using deep generative models, offering new ways to probe how the brain represents real-world scenes. However, many existing approaches first map brain signals into intermediate image or text feature spaces before guiding the generative process, which obscures the contributions of different brain areas to the final reconstruction output. In this work, we propose NeuroAdapter, a visual decoding framework that directly conditions a latent diffusion model on brain representations, bypassing the need for intermediate feature spaces. Our method demonstrates competitive visual reconstruction quality on public fMRI datasets compared to prior work, while providing greater transparency into how brain signals drive visual reconstruction. To this end, we introduce an Image-Brain BI-directional interpretability framework (IBBI) that analyzes cross-attention patterns across diffusion denoising steps to reveal how different cortical areas influence the unfolding generative trajectory. Our work highlights the potential of end-to-end brain-to-image reconstruction and establishes a path for interpretable neural decoding. |
| title | Towards Interpretable Visual Decoding with Attention to Brain Representations |
| topic | Computer Vision and Pattern Recognition 68T07 I.2.0; I.4.9 |
| url | https://arxiv.org/abs/2509.23566 |