Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908373273804800 |
|---|---|
| author | Khanday, Owais Mujtaba Esteban, Pablo Rodroguez San Lone, Zubair Ahmad Ouellet, Marc Lopez, Jose Andres Gonzalez |
| author_facet | Khanday, Owais Mujtaba Esteban, Pablo Rodroguez San Lone, Zubair Ahmad Ouellet, Marc Lopez, Jose Andres Gonzalez |
| contents | Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and speech models can effectively reconstruct high-gamma neural activity characteristics, key indicators of cortical processing, recorded during speech production. We leverage pre-trained embeddings from deep learning models trained on linguistic and acoustic data to represent high-level speech features and map them onto these high-gamma signals. We analyze the extent to which these embeddings preserve the spatio-temporal dynamics of brain activity. Reconstructed neural signals are evaluated against high-gamma ground-truth activity using correlation metrics and signal reconstruction quality assessments. The results indicate that high-gamma activity can be effectively reconstructed using large language and speech model embeddings in all study participants, generating Pearson's correlation coefficients ranging from 0.79 to 0.99. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14074 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings Khanday, Owais Mujtaba Esteban, Pablo Rodroguez San Lone, Zubair Ahmad Ouellet, Marc Lopez, Jose Andres Gonzalez Human-Computer Interaction Sound Audio and Speech Processing Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and speech models can effectively reconstruct high-gamma neural activity characteristics, key indicators of cortical processing, recorded during speech production. We leverage pre-trained embeddings from deep learning models trained on linguistic and acoustic data to represent high-level speech features and map them onto these high-gamma signals. We analyze the extent to which these embeddings preserve the spatio-temporal dynamics of brain activity. Reconstructed neural signals are evaluated against high-gamma ground-truth activity using correlation metrics and signal reconstruction quality assessments. The results indicate that high-gamma activity can be effectively reconstructed using large language and speech model embeddings in all study participants, generating Pearson's correlation coefficients ranging from 0.79 to 0.99. |
| title | Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings |
| topic | Human-Computer Interaction Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.14074 |