BrainLLM: Generative Language Decoding from Brain Recordings
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918182697041920 |
|---|---|
| author | Ye, Ziyi Ai, Qingyao Liu, Yiqun de Rijke, Maarten Zhang, Min Lioma, Christina Ruotsalo, Tuukka |
| author_facet | Ye, Ziyi Ai, Qingyao Liu, Yiqun de Rijke, Maarten Zhang, Min Lioma, Christina Ruotsalo, Tuukka |
| contents | Generating human language through non-invasive brain-computer interfaces (BCIs) has the potential to unlock many applications, such as serving disabled patients and improving communication. Currently, however, generating language via BCIs has been previously successful only within a classification setup for selecting pre-generated sentence continuation candidates with the most likely cortical semantic representation. Inspired by recent research that revealed associations between the brain and the large computational language models, we propose a generative language BCI that utilizes the capacity of a large language model (LLM) jointly with a semantic brain decoder to directly generate language from functional magnetic resonance imaging (fMRI) input. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli perceived, without prior knowledge of any pre-generated candidates. We compare the language generated from the presented model with a random control, pre-generated language selection approach, and a standard LLM, which generates common coherent text solely based on the next word likelihood according to statistical language training data. The proposed model is found to generate language that is more aligned with semantic stimulus in response to which brain input is sampled. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09889 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | BrainLLM: Generative Language Decoding from Brain Recordings Ye, Ziyi Ai, Qingyao Liu, Yiqun de Rijke, Maarten Zhang, Min Lioma, Christina Ruotsalo, Tuukka Computation and Language Generating human language through non-invasive brain-computer interfaces (BCIs) has the potential to unlock many applications, such as serving disabled patients and improving communication. Currently, however, generating language via BCIs has been previously successful only within a classification setup for selecting pre-generated sentence continuation candidates with the most likely cortical semantic representation. Inspired by recent research that revealed associations between the brain and the large computational language models, we propose a generative language BCI that utilizes the capacity of a large language model (LLM) jointly with a semantic brain decoder to directly generate language from functional magnetic resonance imaging (fMRI) input. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli perceived, without prior knowledge of any pre-generated candidates. We compare the language generated from the presented model with a random control, pre-generated language selection approach, and a standard LLM, which generates common coherent text solely based on the next word likelihood according to statistical language training data. The proposed model is found to generate language that is more aligned with semantic stimulus in response to which brain input is sampled. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation. |
| title | BrainLLM: Generative Language Decoding from Brain Recordings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09889 |