Copertina

HSR-Enhanced Sparse Attention Acceleration

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Chen, Bo, Liang, Yingyu, Sha, Zhizhou, Shi, Zhenmei, Song, Zhao
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
di: Cao, Yang, et al.
Pubblicazione: (2024)

Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

A Tighter Complexity Analysis of SparseGPT
di: Li, Xiaoyu, et al.
Pubblicazione: (2024)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

RichSpace: Enriching Text-to-Video Prompt Space via Text Embedding Interpolation
di: Cao, Yuefan, et al.
Pubblicazione: (2025)

HOFAR: High-Order Augmentation of Flow Autoregressive Transformers
di: Liang, Yingyu, et al.
Pubblicazione: (2025)

RoPE Attention Can Be Trained in Almost Linear Time
di: Cao, Yang, et al.
Pubblicazione: (2024)

Towards Infinite-Long Prefix in Transformer
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
di: Li, Xiaoyu, et al.
Pubblicazione: (2024)

On Fine-Grained I/O Complexity of Attention Backward Passes
di: Li, Xiaoyu, et al.
Pubblicazione: (2024)

Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
di: Xu, Zhuoyan, et al.
Pubblicazione: (2024)

Differential Privacy Mechanisms in Neural Tangent Kernel Regression
di: Gu, Jiuxiang, et al.
Pubblicazione: (2024)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
di: Chen, Bo, et al.
Pubblicazione: (2024)

The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
di: Chen, Yifang, et al.
Pubblicazione: (2024)

On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis
di: Ke, Yekun, et al.
Pubblicazione: (2025)

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
di: Shi, Zhenmei, et al.
Pubblicazione: (2024)

Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers
di: Huang, Zekai, et al.
Pubblicazione: (2025)

Circuit Complexity Bounds for Visual Autoregressive Model
di: Ke, Yekun, et al.
Pubblicazione: (2025)

Why Larger Language Models Do In-context Learning Differently?
di: Shi, Zhenmei, et al.
Pubblicazione: (2024)

Universal Approximation of Visual Autoregressive Transformers
di: Chen, Yifang, et al.
Pubblicazione: (2025)

High-Order Matching for One-Step Shortcut Diffusion Models
di: Chen, Bo, et al.
Pubblicazione: (2025)

Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modeling
di: Cao, Yang, et al.
Pubblicazione: (2025)

Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
di: Chen, Yifang, et al.
Pubblicazione: (2025)

Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies
di: Cao, Yuefan, et al.
Pubblicazione: (2025)

Modern Hopfield Networks Require Chain-of-Thought to Solve $\mathsf{NC}^1$-Hard Problems
di: Cao, Yang, et al.
Pubblicazione: (2024)

Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning
di: Xu, Zhuoyan, et al.
Pubblicazione: (2024)

When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time?
di: Li, Chenyang, et al.
Pubblicazione: (2025)

On Computational Limits of FlowAR Models: Expressivity and Efficiency
di: Cao, Yang, et al.
Pubblicazione: (2025)

Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
di: Chen, Bo, et al.
Pubblicazione: (2024)

Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
di: Ke, Yekun, et al.
Pubblicazione: (2024)

Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
di: Cao, Yang, et al.
Pubblicazione: (2024)

Provable Differentially Private Computation of the Cross-Attention Mechanism
di: Ke, Yekun, et al.
Pubblicazione: (2024)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
di: Deng, Yichuan, et al.
Pubblicazione: (2024)

Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
di: Liang, Yingyu, et al.
Pubblicazione: (2024)

On the Computational Capability of Graph Neural Networks: A Circuit Complexity Bound Perspective
di: Li, Xiaoyu, et al.
Pubblicazione: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
di: Yuan, Jingyang, et al.
Pubblicazione: (2025)

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention
di: Zhu, Qianchao, et al.
Pubblicazione: (2024)

Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches
di: Chen, Yifang, et al.
Pubblicazione: (2025)