Brain2Music: Reconstructing Music from Human Brain Activity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917264931946496 |
|---|---|
| author | Denk, Timo I. Takagi, Yu Matsuyama, Takuya Agostinelli, Andrea Nakai, Tomoya Frank, Christian Nishimoto, Shinji |
| author_facet | Denk, Timo I. Takagi, Yu Matsuyama, Takuya Agostinelli, Andrea Nakai, Tomoya Frank, Christian Nishimoto, Shinji |
| contents | The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_11078 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Brain2Music: Reconstructing Music from Human Brain Activity Denk, Timo I. Takagi, Yu Matsuyama, Takuya Agostinelli, Andrea Nakai, Tomoya Frank, Christian Nishimoto, Shinji Neurons and Cognition Machine Learning Sound Audio and Speech Processing The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music |
| title | Brain2Music: Reconstructing Music from Human Brain Activity |
| topic | Neurons and Cognition Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2307.11078 |