Cover Image

On the Role of Depth in the Expressivity of RNNs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lizaire, Maude, Rizvi-Martel, Michael, Dupuis, Éric, Rabusseau, Guillaume
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Tensor Decomposition Perspective on Second-order RNNs
by: Lizaire, Maude, et al.
Published: (2024)

Simulating Weighted Automata over Sequences and Trees with Transformers
by: Rizvi, Michael, et al.
Published: (2024)

The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
by: Rizvi-Martel, Michael, et al.
Published: (2026)

FlowQ-Net: A Generative Framework for Automated Quantum Circuit Design
by: Dai, Jun, et al.
Published: (2025)

Numerical PDE solvers outperform neural PDE solvers
by: Chatain, Patrick, et al.
Published: (2025)

Benefits and Limitations of Communication in Multi-Agent Reasoning
by: Rizvi-Martel, Michael, et al.
Published: (2025)

Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
by: Schöne, Mark, et al.
Published: (2025)

Tensor Cookbook: Mastering Tensors through Diagrams
by: Rakhshan, Beheshteh T., et al.
Published: (2026)

TN-SHAP-G: Graph-Structured Tensor Network Surrogates for Shapley Values and Interactions
by: Heidari, Farzaneh, et al.
Published: (2026)

Tractable Shapley Values and Interactions via Tensor Networks
by: Heidari, Farzaneh, et al.
Published: (2025)

Efficient Probabilistic Tensor Networks
by: Hameed, Marawan Gamal Abdel, et al.
Published: (2025)

KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
by: Lesens, Damien, et al.
Published: (2025)

Learning to (Learn at Test Time): RNNs with Expressive Hidden States
by: Sun, Yu, et al.
Published: (2024)

The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest
by: Lyu, Shen-Huan, et al.
Published: (2024)

Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data
by: Omranpour, Soroush, et al.
Published: (2024)

Higher-Order Transformers With Kronecker-Structured Attention
by: Omranpour, Soroush, et al.
Published: (2024)

Grokking Finite-Dimensional Algebra
by: Notsawo, Pascal Jr Tikeng, et al.
Published: (2026)

Grokking Beyond the Euclidean Norm of Model Parameters
by: Notsawo, Pascal Jr Tikeng, et al.
Published: (2025)

T-GRAB: A Synthetic Diagnostic Benchmark for Learning on Temporal Graphs
by: Dizaji, Alireza, et al.
Published: (2025)

The Effect of Depth on the Expressivity of Deep Linear State-Space Models
by: Bao, Zeyu, et al.
Published: (2025)

Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
by: Brandoit, Julien, et al.
Published: (2026)

UTG: Towards a Unified View of Snapshot and Event Based Models for Temporal Graphs
by: Huang, Shenyang, et al.
Published: (2024)

Memory Caching: RNNs with Growing Memory
by: Behrouz, Ali, et al.
Published: (2026)

Were RNNs All We Needed?
by: Feng, Leo, et al.
Published: (2024)

ROSA: Random Subspace Adaptation for Efficient Fine-Tuning
by: Hameed, Marawan Gamal Abdel, et al.
Published: (2024)

Mechanistic Interpretability of RNNs emulating Hidden Markov Models
by: Torre, Elia, et al.
Published: (2025)

Fixed-Point RNNs: Interpolating from Diagonal to Dense
by: Movahedi, Sajad, et al.
Published: (2025)

Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometry
by: Su, Ye, et al.
Published: (2026)

Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
by: London, Charles, et al.
Published: (2025)

On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth
by: Averkov, Gennadiy, et al.
Published: (2025)

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
by: Merrill, William, et al.
Published: (2025)

Model Merging via Data-Free Covariance Estimation
by: Hameed, Marawan Gamal Abdel, et al.
Published: (2026)

GFlowNets for Hamiltonian decomposition in groups of compatible operators
by: Huidobro-Meezs, Isaac L., et al.
Published: (2024)

ClustRecNet: A Novel End-to-End Deep Learning Framework for Clustering Algorithm Recommendation
by: Bakhtyari, Mohammadreza, et al.
Published: (2025)

Can Local Representation Alignment RNNs Solve Temporal Tasks?
by: Manchev, Nikolay, et al.
Published: (2025)

$\texttt{lrnnx}$: A library for Linear RNNs
by: Bania, Karan, et al.
Published: (2026)

Pessimistic Iterative Planning with RNNs for Robust POMDPs
by: Galesloot, Maris F. L., et al.
Published: (2024)

Paradoxical noise preference in RNNs
by: Eckstein, Noah, et al.
Published: (2026)

On Efficiently Representing Regular Languages as RNNs
by: Svete, Anej, et al.
Published: (2024)

Does Transformer Interpretability Transfer to RNNs?
by: Paulo, Gonçalo, et al.
Published: (2024)