Cover Image

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Xiaoyu, Liang, Yingyu, Shi, Zhenmei, Song, Zhao, Wan, Mingda
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computational Complexity Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Circuit Complexity Bounds for RoPE-based Transformer Architecture
by: Chen, Bo, et al.
Published: (2024)

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Modern Hopfield Networks Require Chain-of-Thought to Solve $\mathsf{NC}^1$-Hard Problems
by: Cao, Yang, et al.
Published: (2024)

Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
by: Chen, Yifang, et al.
Published: (2025)

On Fine-Grained I/O Complexity of Attention Backward Passes
by: Li, Xiaoyu, et al.
Published: (2024)

The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
by: Chen, Yifang, et al.
Published: (2024)

Circuit Complexity Bounds for Visual Autoregressive Model
by: Ke, Yekun, et al.
Published: (2025)

Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers
by: Huang, Zekai, et al.
Published: (2025)

On Computational Limits of FlowAR Models: Expressivity and Efficiency
by: Cao, Yang, et al.
Published: (2025)

When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time?
by: Li, Chenyang, et al.
Published: (2025)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

On the Computational Capability of Graph Neural Networks: A Circuit Complexity Bound Perspective
by: Li, Xiaoyu, et al.
Published: (2025)

Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
by: Liang, Yingyu, et al.
Published: (2024)

On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis
by: Ke, Yekun, et al.
Published: (2025)

Scaling Laws of RoPE-based Extrapolation
by: Liu, Xiaoran, et al.
Published: (2023)

Demystifying the Slash Pattern in Attention: The Role of RoPE
by: Cheng, Yuan, et al.
Published: (2026)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
by: Yang, Ning, et al.
Published: (2026)

Periodic RoPE for Infinite Context LLMs
by: Huo, Simin
Published: (2026)

Universal Approximation of Visual Autoregressive Transformers
by: Chen, Yifang, et al.
Published: (2025)

Towards Infinite-Long Prefix in Transformer
by: Liang, Yingyu, et al.
Published: (2024)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
by: Liu, Feilong
Published: (2026)

HOFAR: High-Order Augmentation of Flow Autoregressive Transformers
by: Liang, Yingyu, et al.
Published: (2025)

A Tighter Complexity Analysis of SparseGPT
by: Li, Xiaoyu, et al.
Published: (2024)

HSR-Enhanced Sparse Attention Acceleration
by: Chen, Bo, et al.
Published: (2024)

The Expressive Power of Transformers with Chain of Thought
by: Merrill, William, et al.
Published: (2023)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
by: Liang, Yingyu, et al.
Published: (2024)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
by: Picov, Isaac, et al.
Published: (2026)

Capturing P: On the Expressive Power and Efficient Evaluation of Boolean Retrieval
by: Aavani, Amir
Published: (2026)

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
by: Brösamle, Moritz, et al.
Published: (2026)

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
by: Liang, Yingyu, et al.
Published: (2024)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
by: Wang, Suyuchen, et al.
Published: (2024)

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
by: Zhong, Meizhi, et al.
Published: (2024)

Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent
by: Chen, Bo, et al.
Published: (2025)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
by: Li, Haoran, et al.
Published: (2026)

Exact Expressive Power of Transformers with Padding
by: Merrill, William, et al.
Published: (2025)

Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
by: Svete, Anej, et al.
Published: (2026)

A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention
by: Ye, Xiaowei, et al.
Published: (2026)