Looking through the mind's eye via multimodal encoder-decoder networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910626454962176 |
|---|---|
| author | Afrasiyabi, Arman Busch, Erica Singh, Rahul Bhaskar, Dhananjay Caplette, Laurent Turk-Browne, Nicholas Krishnaswamy, Smita |
| author_facet | Afrasiyabi, Arman Busch, Erica Singh, Rahul Bhaskar, Dhananjay Caplette, Laurent Turk-Browne, Nicholas Krishnaswamy, Smita |
| contents | In this work, we explore the decoding of mental imagery from subjects using their fMRI measurements. In order to achieve this decoding, we first created a mapping between a subject's fMRI signals elicited by the videos the subjects watched. This mapping associates the high dimensional fMRI activation states with visual imagery. Next, we prompted the subjects textually, primarily with emotion labels which had no direct reference to visual objects. Then to decode visual imagery that may have been in a person's mind's eye, we align a latent representation of these fMRI measurements with a corresponding video-fMRI based on textual labels given to the videos themselves. This alignment has the effect of overlapping the video fMRI embedding with the text-prompted fMRI embedding, thus allowing us to use our fMRI-to-video mapping to decode. Additionally, we enhance an existing fMRI dataset, initially consisting of data from five subjects, by including recordings from three more subjects gathered by our team. We demonstrate the efficacy of our model on this augmented dataset both in accurately creating a mapping, as well as in plausibly decoding mental imagery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00047 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Looking through the mind's eye via multimodal encoder-decoder networks Afrasiyabi, Arman Busch, Erica Singh, Rahul Bhaskar, Dhananjay Caplette, Laurent Turk-Browne, Nicholas Krishnaswamy, Smita Image and Video Processing Machine Learning Neurons and Cognition In this work, we explore the decoding of mental imagery from subjects using their fMRI measurements. In order to achieve this decoding, we first created a mapping between a subject's fMRI signals elicited by the videos the subjects watched. This mapping associates the high dimensional fMRI activation states with visual imagery. Next, we prompted the subjects textually, primarily with emotion labels which had no direct reference to visual objects. Then to decode visual imagery that may have been in a person's mind's eye, we align a latent representation of these fMRI measurements with a corresponding video-fMRI based on textual labels given to the videos themselves. This alignment has the effect of overlapping the video fMRI embedding with the text-prompted fMRI embedding, thus allowing us to use our fMRI-to-video mapping to decode. Additionally, we enhance an existing fMRI dataset, initially consisting of data from five subjects, by including recordings from three more subjects gathered by our team. We demonstrate the efficacy of our model on this augmented dataset both in accurately creating a mapping, as well as in plausibly decoding mental imagery. |
| title | Looking through the mind's eye via multimodal encoder-decoder networks |
| topic | Image and Video Processing Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2410.00047 |