Latent Representation Learning for Multimodal Brain Activity Translation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910622838423552 |
|---|---|
| author | Afrasiyabi, Arman Bhaskar, Dhananjay Busch, Erica L. Caplette, Laurent Singh, Rahul Lajoie, Guillaume Turk-Browne, Nicholas B. Krishnaswamy, Smita |
| author_facet | Afrasiyabi, Arman Bhaskar, Dhananjay Busch, Erica L. Caplette, Laurent Singh, Rahul Lajoie, Guillaume Turk-Browne, Nicholas B. Krishnaswamy, Smita |
| contents | Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18462 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Latent Representation Learning for Multimodal Brain Activity Translation Afrasiyabi, Arman Bhaskar, Dhananjay Busch, Erica L. Caplette, Laurent Singh, Rahul Lajoie, Guillaume Turk-Browne, Nicholas B. Krishnaswamy, Smita Machine Learning Neurons and Cognition Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts. |
| title | Latent Representation Learning for Multimodal Brain Activity Translation |
| topic | Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2409.18462 |