Syntactic Learnability of Echo State Neural Language Models at Scale
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916640993574912 |
|---|---|
| author | Ueda, Ryo Kuribayashi, Tatsuki Kando, Shunsuke Inui, Kentaro |
| author_facet | Ueda, Ryo Kuribayashi, Tatsuki Kando, Shunsuke Inui, Kentaro |
| contents | What is a neural model with minimum architectural complexity that exhibits reasonable language learning capability? To explore such a simple but sufficient neural language model, we revisit a basic reservoir computing (RC) model, Echo State Network (ESN), a restricted class of simple Recurrent Neural Networks. Our experiments showed that ESN with a large hidden state is comparable or superior to Transformer in grammaticality judgment tasks when trained with about 100M words, suggesting that architectures as complex as that of Transformer may not always be necessary for syntactic learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01724 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Syntactic Learnability of Echo State Neural Language Models at Scale Ueda, Ryo Kuribayashi, Tatsuki Kando, Shunsuke Inui, Kentaro Computation and Language What is a neural model with minimum architectural complexity that exhibits reasonable language learning capability? To explore such a simple but sufficient neural language model, we revisit a basic reservoir computing (RC) model, Echo State Network (ESN), a restricted class of simple Recurrent Neural Networks. Our experiments showed that ESN with a large hidden state is comparable or superior to Transformer in grammaticality judgment tasks when trained with about 100M words, suggesting that architectures as complex as that of Transformer may not always be necessary for syntactic learning. |
| title | Syntactic Learnability of Echo State Neural Language Models at Scale |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.01724 |