Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911930876166144 |
|---|---|
| author | Chen, Chi-Sheng Wei, Chun-Shu |
| author_facet | Chen, Chi-Sheng Wei, Chun-Shu |
| contents | Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16910 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning Chen, Chi-Sheng Wei, Chun-Shu Signal Processing Artificial Intelligence Human-Computer Interaction Machine Learning Neurons and Cognition Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG. |
| title | Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning |
| topic | Signal Processing Artificial Intelligence Human-Computer Interaction Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2406.16910 |