Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910888671313920 |
|---|---|
| author | Li, Yueyang Kang, Zijian Gong, Shengyu Dong, Wenhao Zeng, Weiming Yan, Hongjie Siok, Wai Ting Wang, Nizhuan |
| author_facet | Li, Yueyang Kang, Zijian Gong, Shengyu Dong, Wenhao Zeng, Weiming Yan, Hongjie Siok, Wai Ting Wang, Nizhuan |
| contents | Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Codes will be available at: https://github.com/NZWANG/Neural-MCRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17337 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding Li, Yueyang Kang, Zijian Gong, Shengyu Dong, Wenhao Zeng, Weiming Yan, Hongjie Siok, Wai Ting Wang, Nizhuan Computer Vision and Pattern Recognition Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Codes will be available at: https://github.com/NZWANG/Neural-MCRL. |
| title | Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.17337 |