Brain-language fusion enables interactive neural readout and in-silico experimentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909973001273344 |
|---|---|
| author | Bosch, Victoria Anthes, Daniel Doerig, Adrien Thorat, Sushrut König, Peter Kietzmann, Tim Christian |
| author_facet | Bosch, Victoria Anthes, Daniel Doerig, Adrien Thorat, Sushrut König, Peter Kietzmann, Tim Christian |
| contents | Large language models (LLMs) have revolutionized human-machine interaction, and have been extended by embedding diverse modalities such as images into a shared language space. Yet, neural decoding has remained constrained by static, non-interactive methods. We introduce CorText, a framework that integrates neural activity directly into the latent space of an LLM, enabling open-ended, natural language interaction with brain data. Trained on fMRI data recorded during viewing of natural scenes, CorText generates accurate image captions and can answer more detailed questions better than controls, while having access to neural data only. We showcase that CorText achieves zero-shot generalization beyond semantic categories seen during training. In-silico microstimulation experiments, which enable counterfactual prompts on brain activity, reveal a consistent, and graded mapping between brain-state and language output. These advances mark a shift from passive decoding toward generative, flexible interfaces between brain activity and language. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23941 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Brain-language fusion enables interactive neural readout and in-silico experimentation Bosch, Victoria Anthes, Daniel Doerig, Adrien Thorat, Sushrut König, Peter Kietzmann, Tim Christian Machine Learning Neurons and Cognition Large language models (LLMs) have revolutionized human-machine interaction, and have been extended by embedding diverse modalities such as images into a shared language space. Yet, neural decoding has remained constrained by static, non-interactive methods. We introduce CorText, a framework that integrates neural activity directly into the latent space of an LLM, enabling open-ended, natural language interaction with brain data. Trained on fMRI data recorded during viewing of natural scenes, CorText generates accurate image captions and can answer more detailed questions better than controls, while having access to neural data only. We showcase that CorText achieves zero-shot generalization beyond semantic categories seen during training. In-silico microstimulation experiments, which enable counterfactual prompts on brain activity, reveal a consistent, and graded mapping between brain-state and language output. These advances mark a shift from passive decoding toward generative, flexible interfaces between brain activity and language. |
| title | Brain-language fusion enables interactive neural readout and in-silico experimentation |
| topic | Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2509.23941 |