Cover Image

Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bombari, Simone, Mondelli, Marco
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
by: Bombari, Simone, et al.
Published: (2023)

A Law of Data Reconstruction for Random Features (and Beyond)
by: Iurada, Leonardo, et al.
Published: (2025)

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization
by: Bombari, Simone, et al.
Published: (2025)

Privacy for Free in the Overparameterized Regime
by: Bombari, Simone, et al.
Published: (2024)

High-Dimensional Private Linear Regression with Optimal Rates
by: Bombari, Simone, et al.
Published: (2025)

Attention with Trained Embeddings Provably Selects Important Tokens
by: Wu, Diyuan, et al.
Published: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
by: He, Zhengfu, et al.
Published: (2025)

Improving Rare Word Translation With Dictionaries and Attention Masking
by: Sible, Kenneth J., et al.
Published: (2024)

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers
by: Xu, Ruichen, et al.
Published: (2026)

LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
by: Souibgui, Mohamed Ali, et al.
Published: (2026)

Paying Attention to Facts: Quantifying the Knowledge Capacity of Attention Layers
by: Wong, Liang Ze
Published: (2025)

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
by: Bai, Yushi, et al.
Published: (2026)

High-Layer Attention Pruning with Rescaling
by: Liu, Songtao, et al.
Published: (2025)

Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
by: Wu, Diyuan, et al.
Published: (2026)

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
by: Tian, Qiwei, et al.
Published: (2025)

Multi-Layer Attention is the Amplifier of Demonstration Effectiveness
by: Wang, Dingzirui, et al.
Published: (2025)

Guided Perturbation Sensitivity (GPS): Detecting Adversarial Text via Embedding Stability and Word Importance
by: Tuck, Bryan E., et al.
Published: (2025)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
by: Jin, Zehao, et al.
Published: (2026)

Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learning
by: Rho, Donghwan
Published: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
by: Monteiro, João, et al.
Published: (2026)

Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions
by: Valentino, Marco, et al.
Published: (2023)

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
by: Bozic, Vukasin, et al.
Published: (2023)

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
by: Zhu, Hanlin, et al.
Published: (2024)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
by: Musat, Tiberiu
Published: (2024)

CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
by: McDanel, Bradley, et al.
Published: (2026)

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
by: Wang, Zihao, et al.
Published: (2024)

Extracting Rule-based Descriptions of Attention Features in Transformers
by: Friedman, Dan, et al.
Published: (2025)

Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers
by: Chen, Lei, et al.
Published: (2024)

Towards Understanding Steering Strength
by: Taimeskhanov, Magamed, et al.
Published: (2026)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
by: Fu, Zizhuo, et al.
Published: (2026)

Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification
by: ElNokrashy, Muhammad, et al.
Published: (2022)

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
by: Brandon, William, et al.
Published: (2024)

Wave-PDE Nets: Trainable Wave-Equation Layers as an Alternative to Attention
by: Vejendla, Harshil
Published: (2025)

Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
by: Karkada, Dhruva, et al.
Published: (2025)

Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
by: Aquino-Michaels, Keston
Published: (2026)

Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
by: Dong, Yihe, et al.
Published: (2025)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
by: Deng, Difan, et al.
Published: (2026)

Decomposing Attention To Find Context-Sensitive Neurons
by: Gibson, Alex
Published: (2025)

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
by: Zarch, Hossein Entezari, et al.
Published: (2025)

Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime
by: Wu, Diyuan, et al.
Published: (2025)