Imagen de Portada

High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Barnfield, Nicholas, Cui, Hugo, Lu, Yue M.
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval
por: Barnfield, Nicholas, et al.
Publicado: (2026)

Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning
por: Barnfield, Nicholas, et al.
Publicado: (2026)

Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning
por: Cui, Hugo, et al.
Publicado: (2026)

Interpreting Attention Layer Outputs with Sparse Autoencoders
por: Kissane, Connor, et al.
Publicado: (2024)

High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates
por: Lu, Fred, et al.
Publicado: (2024)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
por: Jo, Dongwon, et al.
Publicado: (2026)

STS: Efficient Sparse Attention with Speculative Token Sparsity
por: Xu, Ceyu, et al.
Publicado: (2026)

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
por: Meng, Weikang, et al.
Publicado: (2026)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
por: He, Mutian, et al.
Publicado: (2025)

Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
por: Sakai, Mana, et al.
Publicado: (2025)

Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Tasks
por: Krishnan, Adit, et al.
Publicado: (2025)

Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
por: Song, Zhuo-Yang, et al.
Publicado: (2025)

Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks
por: Rangriz, Parsa
Publicado: (2025)

Neighbor Embedding for High-Dimensional Sparse Poisson Data
por: Mudrik, Noga, et al.
Publicado: (2026)

Hierarchical Sparse Representation Clustering for High-Dimensional Data Streams
por: Chen, Jie, et al.
Publicado: (2024)

Gradient Boosting within a Single Attention Layer
por: Sargolzaei, Saleh
Publicado: (2026)

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
por: Zarch, Hossein Entezari, et al.
Publicado: (2025)

Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning
por: Bond, Andrew, et al.
Publicado: (2024)

High-Layer Attention Pruning with Rescaling
por: Liu, Songtao, et al.
Publicado: (2025)

Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
por: Wang, Junxuan, et al.
Publicado: (2025)

A solvable model of learning generative diffusion: theory and insights
por: Cui, Hugo, et al.
Publicado: (2025)

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
por: Bai, Yushi, et al.
Publicado: (2026)

Token Sample Complexity of Attention
por: Bohbot, Léa, et al.
Publicado: (2025)

Geometric Analysis of Token Selection in Multi-Head Attention
por: Mudarisov, Timur, et al.
Publicado: (2026)

Virtual Node Generation for Node Classification in Sparsely-Labeled Graphs
por: Cui, Hang, et al.
Publicado: (2024)

AdaSplash-2: Faster Differentiable Sparse Attention
por: Gonçalves, Nuno, et al.
Publicado: (2026)

Learning to Predict, Discover, and Reason in High-Dimensional Event Sequences
por: Math, Hugo
Publicado: (2026)

ToMA: Token Merge with Attention for Diffusion Models
por: Lu, Wenbo, et al.
Publicado: (2025)

Hardness of High-Dimensional Linear Classification
por: Munteanu, Alexander, et al.
Publicado: (2026)

MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation
por: Behrendt, Maike, et al.
Publicado: (2025)

Sparse Autoencoder Features for Classifications and Transferability
por: Gallifant, Jack, et al.
Publicado: (2025)

Solving Sparse \& High-Dimensional-Output Regression via Compression
por: Li, Renyuan, et al.
Publicado: (2024)

High-dimensional learning of narrow neural networks
por: Cui, Hugo
Publicado: (2024)

Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks
por: Arnaboldi, Luca, et al.
Publicado: (2025)

vAttention: Verified Sparse Attention
por: Desai, Aditya, et al.
Publicado: (2025)

Improving Routing in Sparse Mixture of Experts with Graph of Tokens
por: Nguyen, Tam, et al.
Publicado: (2025)

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
por: Wang, Hanrui, et al.
Publicado: (2020)

AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity
por: Zhang, Yu, et al.
Publicado: (2025)

A Convergence Analysis of Approximate Message Passing with Non-Separable Functions and Applications to Multi-Class Classification
por: Çakmak, Burak, et al.
Publicado: (2024)

MambaSL: Exploring Single-Layer Mamba for Time Series Classification
por: Jung, Yoo-Min, et al.
Publicado: (2026)