Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.18417 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910028231868416 |
|---|---|
| author | Nunley, Joshua |
| author_facet | Nunley, Joshua |
| contents | This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d). We use a minimal axiomatic setup and derive recurrent and transformer templates from a shared skeleton in which subgroup choice acts as a drop-in replacement for state space, tangent projection, and update map. We then specialize to O(d) and evaluate orthogonal-state RNN and transformer models on Tiny Shakespeare and Penn Treebank under parameter-matched settings. We also report a general linear-mixing extension in tangent space, which applies across subgroup choices and improves finite-budget performance in the current O(d) experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18417 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Subgroups of $U(d)$ Induce Natural RNN and Transformer Architectures Nunley, Joshua Machine Learning Computation and Language 68T07, 22E70 I.2.6; G.3 This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d). We use a minimal axiomatic setup and derive recurrent and transformer templates from a shared skeleton in which subgroup choice acts as a drop-in replacement for state space, tangent projection, and update map. We then specialize to O(d) and evaluate orthogonal-state RNN and transformer models on Tiny Shakespeare and Penn Treebank under parameter-matched settings. We also report a general linear-mixing extension in tangent space, which applies across subgroup choices and improves finite-budget performance in the current O(d) experiments. |
| title | Subgroups of $U(d)$ Induce Natural RNN and Transformer Architectures |
| topic | Machine Learning Computation and Language 68T07, 22E70 I.2.6; G.3 |
| url | https://arxiv.org/abs/2602.18417 |