Toward Fully-End-to-End Listened Speech Decoding from EEG Signals
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911915598413824 |
|---|---|
| author | Lee, Jihwan Kommineni, Aditya Feng, Tiantian Avramidis, Kleanthis Shi, Xuan Kadiri, Sudarsana Narayanan, Shrikanth |
| author_facet | Lee, Jihwan Kommineni, Aditya Feng, Tiantian Avramidis, Kleanthis Shi, Xuan Kadiri, Sudarsana Narayanan, Shrikanth |
| contents | Speech decoding from EEG signals is a challenging task, where brain activity is modeled to estimate salient characteristics of acoustic stimuli. We propose FESDE, a novel framework for Fully-End-to-end Speech Decoding from EEG signals. Our approach aims to directly reconstruct listened speech waveforms given EEG signals, where no intermediate acoustic feature processing step is required. The proposed method consists of an EEG module and a speech module along with a connector. The EEG module learns to better represent EEG signals, while the speech module generates speech waveforms from model representations. The connector learns to bridge the distributions of the latent spaces of EEG and speech. The proposed framework is both simple and efficient, by allowing single-step inference, and outperforms prior works on objective metrics. A fine-grained phoneme analysis is conducted to unveil model characteristics of speech decoding. The source code is available here: github.com/lee-jhwn/fesde. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Toward Fully-End-to-End Listened Speech Decoding from EEG Signals Lee, Jihwan Kommineni, Aditya Feng, Tiantian Avramidis, Kleanthis Shi, Xuan Kadiri, Sudarsana Narayanan, Shrikanth Signal Processing Artificial Intelligence Sound Audio and Speech Processing Speech decoding from EEG signals is a challenging task, where brain activity is modeled to estimate salient characteristics of acoustic stimuli. We propose FESDE, a novel framework for Fully-End-to-end Speech Decoding from EEG signals. Our approach aims to directly reconstruct listened speech waveforms given EEG signals, where no intermediate acoustic feature processing step is required. The proposed method consists of an EEG module and a speech module along with a connector. The EEG module learns to better represent EEG signals, while the speech module generates speech waveforms from model representations. The connector learns to bridge the distributions of the latent spaces of EEG and speech. The proposed framework is both simple and efficient, by allowing single-step inference, and outperforms prior works on objective metrics. A fine-grained phoneme analysis is conducted to unveil model characteristics of speech decoding. The source code is available here: github.com/lee-jhwn/fesde. |
| title | Toward Fully-End-to-End Listened Speech Decoding from EEG Signals |
| topic | Signal Processing Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.08644 |