RealMind: Advancing Visual Decoding and Language Interaction via EEG Signals
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909443180986368 |
|---|---|
| author | Li, Dongyang Qin, Haoyang Wu, Mingyang Tang, Jiahua Cao, Yuang Wei, Chen Liu, Quanying |
| author_facet | Li, Dongyang Qin, Haoyang Wu, Mingyang Tang, Jiahua Cao, Yuang Wei, Chen Liu, Quanying |
| contents | Decoding visual stimuli from neural recordings is a critical challenge in the development of brain-computer interfaces (BCIs). Although recent EEG-based decoding approaches have made progress in tasks such as visual classification, retrieval, and reconstruction, they remain constrained by unstable representation learning and a lack of interpretability. This gap highlights the need for more efficient representation learning and the integration of effective language interaction to enhance both understanding and practical usability in visual decoding tasks.To address this limitation, we introduce RealMind, a novel EEG-based framework designed to handle a diverse range of downstream tasks. Specifically, RealMind leverages both semantic and geometric consistency learning to enhance feature representation and improve alignment across tasks. Notably, beyond excelling in traditional tasks, our framework marks the first attempt at visual captioning from EEG data through vision-language model (VLM). It achieves a Top-1 decoding accuracy of 27.58% in a 200-class zero-shot retrieval task and a BLEU-1 score of 26.59% in a 200-class zero-shot captioning task. Overall, RealMind provides a comprehensive multitask EEG decoding framework, establishing a foundational approach for EEG-based visual decoding in real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23754 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RealMind: Advancing Visual Decoding and Language Interaction via EEG Signals Li, Dongyang Qin, Haoyang Wu, Mingyang Tang, Jiahua Cao, Yuang Wei, Chen Liu, Quanying Human-Computer Interaction Neurons and Cognition Decoding visual stimuli from neural recordings is a critical challenge in the development of brain-computer interfaces (BCIs). Although recent EEG-based decoding approaches have made progress in tasks such as visual classification, retrieval, and reconstruction, they remain constrained by unstable representation learning and a lack of interpretability. This gap highlights the need for more efficient representation learning and the integration of effective language interaction to enhance both understanding and practical usability in visual decoding tasks.To address this limitation, we introduce RealMind, a novel EEG-based framework designed to handle a diverse range of downstream tasks. Specifically, RealMind leverages both semantic and geometric consistency learning to enhance feature representation and improve alignment across tasks. Notably, beyond excelling in traditional tasks, our framework marks the first attempt at visual captioning from EEG data through vision-language model (VLM). It achieves a Top-1 decoding accuracy of 27.58% in a 200-class zero-shot retrieval task and a BLEU-1 score of 26.59% in a 200-class zero-shot captioning task. Overall, RealMind provides a comprehensive multitask EEG decoding framework, establishing a foundational approach for EEG-based visual decoding in real-world applications. |
| title | RealMind: Advancing Visual Decoding and Language Interaction via EEG Signals |
| topic | Human-Computer Interaction Neurons and Cognition |
| url | https://arxiv.org/abs/2410.23754 |