Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Language: | En |
| Published: |
Zenodo
2026
|
| Subjects: | |
| Online Access: | https://doi.org/10.5281/zenodo.18903774 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Table of Contents:
- <p>We present two hybrid architectures that combine Echo State Network (ESN) reservoirs with transformer-style attention for parameter-efficient character-level language modelling. <em>Fixed-KV Attention</em> replaces learned K/V projections by fixed random linear maps of reservoir states, while <em>Node Attention</em> reframes attention as a per-step, query-gated readout over reservoir nodes, reducing attention complexity from O(T²) to O(R) per step. On TinyShakespeare, Node Attention achieves the best validation loss (1.969), beating an off-the-shelf transformer and the AERC literature approach while training at ≈21.8k tokens/s on CPU and using only 347k trained parameters. Our results show that rich reservoir dynamics plus a query-gated node readout provide a compact, efficient alternative to learned K/V projections and point to a practical route for long-context, efficient language modelling.</p>