Image de couverture de livre

On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Xu, Kevin, Sato, Issei
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers
par: Xu, Kevin, et autres
Publié: (2025)

A Formal Comparison Between Chain of Thought and Latent Thought
par: Xu, Kevin, et autres
Publié: (2025)

Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
par: Tomihari, Akiyoshi, et autres
Publié: (2026)

Understanding Transformer Optimization via Gradient Heterogeneity
par: Tomihari, Akiyoshi, et autres
Publié: (2025)

Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective
par: Koshizuka, Takeshi, et autres
Publié: (2023)

On the Optimal Memorization Capacity of Transformers
par: Kajitsuka, Tokio, et autres
Publié: (2024)

End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
par: Sakamoto, Keitaro, et autres
Publié: (2024)

Can Test-time Computation Mitigate Reproduction Bias in Neural Symbolic Regression?
par: Sato, Shun, et autres
Publié: (2025)

Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding
par: Hayakawa, Daichi, et autres
Publié: (2024)

Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
par: Li, Yin
Publié: (2025)

Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
par: Kajitsuka, Tokio, et autres
Publié: (2023)

Benign Overfitting in Token Selection of Attention Mechanism
par: Sakamoto, Keitaro, et autres
Publié: (2024)

Multiplicative Logit Adjustment Approximates Neural-Collapse-Aware Decision Boundary Adjustment
par: Hasegawa, Naoya, et autres
Publié: (2024)

Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
par: Tomihari, Akiyoshi, et autres
Publié: (2024)

Top-Down Bayesian Posterior Sampling for Sum-Product Networks
par: Yokoi, Soma, et autres
Publié: (2024)

Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
par: Sakamoto, Keitaro, et autres
Publié: (2025)

Exploring Weight Balancing on Long-Tailed Recognition Problem
par: Hasegawa, Naoya, et autres
Publié: (2023)

A Theoretical Study of Neural Network Expressive Power via Manifold Topology
par: Yao, Jiachen, et autres
Publié: (2024)

On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformers
par: Zhou, Cai, et autres
Publié: (2024)

On the Expressive Power of Floating-Point Transformers
par: Park, Sejun, et autres
Publié: (2026)

On the Expressive Power of Contextual Relations in Transformers
par: Fraiman, Demián
Publié: (2026)

Rethinking Associative Memory Mechanism in Induction Head
par: Wang, Shuo, et autres
Publié: (2024)

Understanding Generalization in Physics Informed Models through Affine Variety Dimensions
par: Koshizuka, Takeshi, et autres
Publié: (2025)

Max-pooling Network Revisited: Analyzing the Role of Semantic Probability in Multiple Instance Learning for Hallucination Detection
par: Fujikawa, Shota, et autres
Publié: (2026)

Fix Initial Codes and Iteratively Refine Textual Directions Toward Safe Multi-Turn Code Correction
par: Tanaka, Yuto, et autres
Publié: (2026)

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
par: Wang, Mingze, et autres
Publié: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
par: Li, Xiaoyu, et autres
Publié: (2024)

Transformers are Expressive, But Are They Expressive Enough for Regression?
par: Nath, Swaroop, et autres
Publié: (2024)

Relational Preference Encoding in Looped Transformer Internal States
par: Kirin, Jan
Publié: (2026)

On the Stability of Expressive Positional Encodings for Graphs
par: Huang, Yinan, et autres
Publié: (2023)

Exact Expressive Power of Transformers with Padding
par: Merrill, William, et autres
Publié: (2025)

Rethinking the Expressive Power of GNNs via Graph Biconnectivity
par: Zhang, Bohang, et autres
Publié: (2023)

The Expressive Power of Transformers with Chain of Thought
par: Merrill, William, et autres
Publié: (2023)

On The Expressive Power of GNN Derivatives
par: Eitan, Yam, et autres
Publié: (2025)

Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
par: Chakrabarti, Sayak, et autres
Publié: (2026)

On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions
par: Gu, Linyan, et autres
Publié: (2026)

Expressive Power of Temporal Message Passing
par: Wałęga, Przemysław Andrzej, et autres
Publié: (2024)

Energy-Entropy Regularization: The True Power of Minimal Looped Transformers
par: Lam, Wai-Lun
Publié: (2026)

Looped Transformers with Layer Normalization Provably Learn the Power Method
par: Wu, Lyumin, et autres
Publié: (2026)

On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
par: Xie, Zeke, et autres
Publié: (2020)