Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912181716516864 |
|---|---|
| author | Lee, Jihwan Feng, Tiantian Kommineni, Aditya Kadiri, Sudarsana Reddy Narayanan, Shrikanth |
| author_facet | Lee, Jihwan Feng, Tiantian Kommineni, Aditya Kadiri, Sudarsana Reddy Narayanan, Shrikanth |
| contents | Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities. The source code and speech samples are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_04844 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction Lee, Jihwan Feng, Tiantian Kommineni, Aditya Kadiri, Sudarsana Reddy Narayanan, Shrikanth Audio and Speech Processing Artificial Intelligence Computation and Language Signal Processing Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities. The source code and speech samples are publicly available. |
| title | Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Signal Processing |
| url | https://arxiv.org/abs/2501.04844 |