Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.14468 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913801008316416 |
|---|---|
| author | Liu, Yijun |
| author_facet | Liu, Yijun |
| contents | Interpreting neural activity through meaningful latent representations remains a complex and evolving challenge at the intersection of neuroscience and artificial intelligence. We investigate the potential of multimodal foundation models to align invasive brain recordings with natural language. We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model, enabling sentence-level retrieval directly from brain activity. SSENSE trains a neural encoder on spectral representations of sEEG using InfoNCE loss, without fine-tuning the text encoder. We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset. Despite limited data, SSENSE achieves promising results, demonstrating that general-purpose language representations can serve as effective priors for neural decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14468 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment Liu, Yijun Computation and Language Machine Learning Signal Processing Neurons and Cognition Interpreting neural activity through meaningful latent representations remains a complex and evolving challenge at the intersection of neuroscience and artificial intelligence. We investigate the potential of multimodal foundation models to align invasive brain recordings with natural language. We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model, enabling sentence-level retrieval directly from brain activity. SSENSE trains a neural encoder on spectral representations of sEEG using InfoNCE loss, without fine-tuning the text encoder. We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset. Despite limited data, SSENSE achieves promising results, demonstrating that general-purpose language representations can serve as effective priors for neural decoding. |
| title | sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment |
| topic | Computation and Language Machine Learning Signal Processing Neurons and Cognition |
| url | https://arxiv.org/abs/2504.14468 |