Brain-aligning of semantic vectors improves neural decoding of visual stimuli
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918226207703040 |
|---|---|
| author | Vafaei, Shirin Fukuma, Ryohei Yanagisawa, Takufumi Yang, Huixiang Oshino, Satoru Tani, Naoki Khoo, Hui Ming Sugano, Hidenori Iimura, Yasushi Suzuki, Hiroharu Nakajima, Madoka Tamura, Kentaro Kishima, Haruhiko |
| author_facet | Vafaei, Shirin Fukuma, Ryohei Yanagisawa, Takufumi Yang, Huixiang Oshino, Satoru Tani, Naoki Khoo, Hui Ming Sugano, Hidenori Iimura, Yasushi Suzuki, Hiroharu Nakajima, Madoka Tamura, Kentaro Kishima, Haruhiko |
| contents | The development of algorithms to accurately decode neural information has long been a research focus in the field of neuroscience. Brain decoding typically involves training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might fundamentally differ from those that are encoded by the brain, limiting the ability of decoders to accurately learn this mapping. To address this issue, we propose a framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the brain. We trained this model with functional magnetic resonance imaging (fMRI) and then performed zero-shot brain decoding on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data. fMRI-based brain-aligned vectors improved decoding performance across all three neuroimaging datasets when accuracy was determined by calculating the correlation coefficients between true and predicted vectors. Additionally, when decoding accuracy was determined via stimulus identification, this accuracy increased in specific category types; improvements varied depending on the original vector space that was used for brain-alignment, and consistent improvements were observed across all neuroimaging modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_15176 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Brain-aligning of semantic vectors improves neural decoding of visual stimuli Vafaei, Shirin Fukuma, Ryohei Yanagisawa, Takufumi Yang, Huixiang Oshino, Satoru Tani, Naoki Khoo, Hui Ming Sugano, Hidenori Iimura, Yasushi Suzuki, Hiroharu Nakajima, Madoka Tamura, Kentaro Kishima, Haruhiko Neurons and Cognition Artificial Intelligence The development of algorithms to accurately decode neural information has long been a research focus in the field of neuroscience. Brain decoding typically involves training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might fundamentally differ from those that are encoded by the brain, limiting the ability of decoders to accurately learn this mapping. To address this issue, we propose a framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the brain. We trained this model with functional magnetic resonance imaging (fMRI) and then performed zero-shot brain decoding on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data. fMRI-based brain-aligned vectors improved decoding performance across all three neuroimaging datasets when accuracy was determined by calculating the correlation coefficients between true and predicted vectors. Additionally, when decoding accuracy was determined via stimulus identification, this accuracy increased in specific category types; improvements varied depending on the original vector space that was used for brain-alignment, and consistent improvements were observed across all neuroimaging modalities. |
| title | Brain-aligning of semantic vectors improves neural decoding of visual stimuli |
| topic | Neurons and Cognition Artificial Intelligence |
| url | https://arxiv.org/abs/2403.15176 |