Are EEG-to-Text Models Working?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929560803606528 |
|---|---|
| author | Jo, Hyejeong Yang, Yiqian Han, Juhyeok Duan, Yiqun Xiong, Hui Lee, Won Hee |
| author_facet | Jo, Hyejeong Yang, Yiqian Han, Juhyeok Duan, Yiqun Xiong, Hui Lee, Won Hee |
| contents | This work critically analyzes existing models for open-vocabulary EEG-to-Text translation. We identify a crucial limitation: previous studies often employed implicit teacher-forcing during evaluation, artificially inflating performance metrics. Additionally, they lacked a critical benchmark - comparing model performance on pure noise inputs. We propose a methodology to differentiate between models that truly learn from EEG signals and those that simply memorize training data. Our analysis reveals that model performance on noise data can be comparable to that on EEG data. These findings highlight the need for stricter evaluation practices in EEG-to-Text research, emphasizing transparent reporting and rigorous benchmarking with noise inputs. This approach will lead to more reliable assessments of model capabilities and pave the way for robust EEG-to-Text communication systems. Code is available at https://github.com/NeuSpeech/EEG-To-Text |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_06459 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are EEG-to-Text Models Working? Jo, Hyejeong Yang, Yiqian Han, Juhyeok Duan, Yiqun Xiong, Hui Lee, Won Hee Computation and Language Artificial Intelligence This work critically analyzes existing models for open-vocabulary EEG-to-Text translation. We identify a crucial limitation: previous studies often employed implicit teacher-forcing during evaluation, artificially inflating performance metrics. Additionally, they lacked a critical benchmark - comparing model performance on pure noise inputs. We propose a methodology to differentiate between models that truly learn from EEG signals and those that simply memorize training data. Our analysis reveals that model performance on noise data can be comparable to that on EEG data. These findings highlight the need for stricter evaluation practices in EEG-to-Text research, emphasizing transparent reporting and rigorous benchmarking with noise inputs. This approach will lead to more reliable assessments of model capabilities and pave the way for robust EEG-to-Text communication systems. Code is available at https://github.com/NeuSpeech/EEG-To-Text |
| title | Are EEG-to-Text Models Working? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.06459 |