DREAM: Visual Decoding from Reversing Human Visual System
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913307965784064 |
|---|---|
| author | Xia, Weihao de Charette, Raoul Öztireli, Cengiz Xue, Jing-Hao |
| author_facet | Xia, Weihao de Charette, Raoul Öztireli, Cengiz Xue, Jing-Hao |
| contents | In this work we present DREAM, an fMRI-to-image method for reconstructing viewed images from brain activities, grounded on fundamental knowledge of the human visual system. We craft reverse pathways that emulate the hierarchical and parallel nature of how humans perceive the visual world. These tailored pathways are specialized to decipher semantics, color, and depth cues from fMRI data, mirroring the forward pathways from visual stimuli to fMRI recordings. To do so, two components mimic the inverse processes within the human visual system: the Reverse Visual Association Cortex (R-VAC) which reverses pathways of this brain region, extracting semantics from fMRI data; the Reverse Parallel PKM (R-PKM) component simultaneously predicting color and depth from fMRI signals. The experiments indicate that our method outperforms the current state-of-the-art models in terms of the consistency of appearance, structure, and semantics. Code will be made publicly available to facilitate further research in this field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02265 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DREAM: Visual Decoding from Reversing Human Visual System Xia, Weihao de Charette, Raoul Öztireli, Cengiz Xue, Jing-Hao Computer Vision and Pattern Recognition Machine Learning Image and Video Processing Neurons and Cognition In this work we present DREAM, an fMRI-to-image method for reconstructing viewed images from brain activities, grounded on fundamental knowledge of the human visual system. We craft reverse pathways that emulate the hierarchical and parallel nature of how humans perceive the visual world. These tailored pathways are specialized to decipher semantics, color, and depth cues from fMRI data, mirroring the forward pathways from visual stimuli to fMRI recordings. To do so, two components mimic the inverse processes within the human visual system: the Reverse Visual Association Cortex (R-VAC) which reverses pathways of this brain region, extracting semantics from fMRI data; the Reverse Parallel PKM (R-PKM) component simultaneously predicting color and depth from fMRI signals. The experiments indicate that our method outperforms the current state-of-the-art models in terms of the consistency of appearance, structure, and semantics. Code will be made publicly available to facilitate further research in this field. |
| title | DREAM: Visual Decoding from Reversing Human Visual System |
| topic | Computer Vision and Pattern Recognition Machine Learning Image and Video Processing Neurons and Cognition |
| url | https://arxiv.org/abs/2310.02265 |