Decoding Continuous Character-based Language from Non-invasive Brain Recordings
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910372982685696 |
|---|---|
| author | Zhang, Cenyuan Zheng, Xiaoqing Yin, Ruicheng Geng, Shujie Xu, Jianhan Gao, Xuan Lv, Changze Ling, Zixuan Huang, Xuanjing Cao, Miao Feng, Jianfeng |
| author_facet | Zhang, Cenyuan Zheng, Xiaoqing Yin, Ruicheng Geng, Shujie Xu, Jianhan Gao, Xuan Lv, Changze Ling, Zixuan Huang, Xuanjing Cao, Miao Feng, Jianfeng |
| contents | Deciphering natural language from brain activity through non-invasive devices remains a formidable challenge. Previous non-invasive decoders either require multiple experiments with identical stimuli to pinpoint cortical regions and enhance signal-to-noise ratios in brain activity, or they are limited to discerning basic linguistic elements such as letters and words. We propose a novel approach to decoding continuous language from single-trial non-invasive fMRI recordings, in which a three-dimensional convolutional network augmented with information bottleneck is developed to automatically identify responsive voxels to stimuli, and a character-based decoder is designed for the semantic reconstruction of continuous language characterized by inherent character structures. The resulting decoder can produce intelligible textual sequences that faithfully capture the meaning of perceived speech both within and across subjects, while existing decoders exhibit significantly inferior performance in cross-subject contexts. The ability to decode continuous language from single trials across subjects demonstrates the promising applications of non-invasive language brain-computer interfaces in both healthcare and neuroscience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_11183 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Decoding Continuous Character-based Language from Non-invasive Brain Recordings Zhang, Cenyuan Zheng, Xiaoqing Yin, Ruicheng Geng, Shujie Xu, Jianhan Gao, Xuan Lv, Changze Ling, Zixuan Huang, Xuanjing Cao, Miao Feng, Jianfeng Computation and Language Deciphering natural language from brain activity through non-invasive devices remains a formidable challenge. Previous non-invasive decoders either require multiple experiments with identical stimuli to pinpoint cortical regions and enhance signal-to-noise ratios in brain activity, or they are limited to discerning basic linguistic elements such as letters and words. We propose a novel approach to decoding continuous language from single-trial non-invasive fMRI recordings, in which a three-dimensional convolutional network augmented with information bottleneck is developed to automatically identify responsive voxels to stimuli, and a character-based decoder is designed for the semantic reconstruction of continuous language characterized by inherent character structures. The resulting decoder can produce intelligible textual sequences that faithfully capture the meaning of perceived speech both within and across subjects, while existing decoders exhibit significantly inferior performance in cross-subject contexts. The ability to decode continuous language from single trials across subjects demonstrates the promising applications of non-invasive language brain-computer interfaces in both healthcare and neuroscience. |
| title | Decoding Continuous Character-based Language from Non-invasive Brain Recordings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.11183 |