Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.07239 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910634031972352 |
|---|---|
| author | Biegun, Kai Dolga, Rares Cunningham, Jake Barber, David |
| author_facet | Biegun, Kai Dolga, Rares Cunningham, Jake Barber, David |
| contents | Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07239 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RotRNN: Modelling Long Sequences with Rotations Biegun, Kai Dolga, Rares Cunningham, Jake Barber, David Machine Learning Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets. |
| title | RotRNN: Modelling Long Sequences with Rotations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.07239 |