Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2605.06729 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915989904424960 |
|---|---|
| author | Shahmansoori, Arash |
| author_facet | Shahmansoori, Arash |
| contents | We present the E$Δ$-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at $β\in \{0,2\}$, our Data-Dependent Cayley rotation $Q(x)=(I+(β/2)A(x))^{-1}(I-(β/2)A(x))$ preserves orthogonality for all $β$ and all inputs. To handle negation, an eigenvalue $-1$ case that Cayley provably excludes, we introduce the E$Δ$-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate $X'=γ(X)Q(X)X+(1-γ(X))H_2(X)X$. A midpoint-collapse regularizer, $4γ(1-γ)$, encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, E$Δ$-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-$π$ rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact $λ=-1$ operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of $O(n)$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06729 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality Shahmansoori, Arash Machine Learning Artificial Intelligence 68T07 (Primary), 15B10 (Secondary) I.2.6; I.5.1; G.1.3 We present the E$Δ$-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at $β\in \{0,2\}$, our Data-Dependent Cayley rotation $Q(x)=(I+(β/2)A(x))^{-1}(I-(β/2)A(x))$ preserves orthogonality for all $β$ and all inputs. To handle negation, an eigenvalue $-1$ case that Cayley provably excludes, we introduce the E$Δ$-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate $X'=γ(X)Q(X)X+(1-γ(X))H_2(X)X$. A midpoint-collapse regularizer, $4γ(1-γ)$, encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, E$Δ$-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-$π$ rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact $λ=-1$ operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of $O(n)$. |
| title | The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality |
| topic | Machine Learning Artificial Intelligence 68T07 (Primary), 15B10 (Secondary) I.2.6; I.5.1; G.1.3 |
| url | https://arxiv.org/abs/2605.06729 |