Cover Image

Linear Predictability of Attention Heads in Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shaikh, Khalid, Singh, Asmit Kumar, Dsouza, Rebecca Christopher, Shiromani, Shikhar
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
by: Mann, Logan, et al.
Published: (2026)

Focus Where It Matters: Graph Selective State Focused Attention Networks
by: Vashistha, Shikhar, et al.
Published: (2024)

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration
by: Yu, Zhongzhi, et al.
Published: (2024)

MHARFedLLM: Multimodal Human Activity Recognition Using Federated Large Language Model
by: Bandyopadhyay, Asmit, et al.
Published: (2025)

Improving Multimodal Large Language Models Using Continual Learning
by: Srivastava, Shikhar, et al.
Published: (2024)

CLAReSNet: When Convolution Meets Latent Attention for Hyperspectral Image Classification
by: Bandyopadhyay, Asmit, et al.
Published: (2025)

On the Role of Attention Heads in Large Language Model Safety
by: Zhou, Zhenhong, et al.
Published: (2024)

Execution-Grounded Credit Assignment for GRPO in Code Generation
by: Kumar, Abhijit, et al.
Published: (2026)

Superiority of Multi-Head Attention in In-Context Linear Regression
by: Cui, Yingqian, et al.
Published: (2024)

Attention Head Entropy of LLMs Predicts Answer Correctness
by: Ostmeier, Sophie, et al.
Published: (2026)

MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
by: Kallini, Julie, et al.
Published: (2024)

Interleaved Head Attention
by: Duvvuri, Sai Surya, et al.
Published: (2026)

RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts
by: Joshi, Sahil, et al.
Published: (2025)

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
by: Erden, Caner
Published: (2025)

In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
by: He, Jianliang, et al.
Published: (2025)

Adaptive Head Budgeting for Efficient Multi-Head Attention
by: Faye, Bilal, et al.
Published: (2026)

Robust and Noise-resilient Long-Term Prediction of Spatiotemporal Data Using Variational Mode Graph Neural Networks with 3D Attention
by: Ahmad, Osama, et al.
Published: (2025)

Fourier Head: Helping Large Language Models Learn Complex Probability Distributions
by: Gillman, Nate, et al.
Published: (2024)

Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
by: Kim, Jinyeong, et al.
Published: (2025)

Prediction of Herd Life in Dairy Cows Using Multi-Head Attention Transformers
by: Saki, Mahdi, et al.
Published: (2025)

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
by: Pandey, Vishal, et al.
Published: (2026)

Training Dynamics of In-Context Learning in Linear Attention
by: Zhang, Yedi, et al.
Published: (2025)

Routoo: Learning to Route to Large Language Models Effectively
by: Mohammadshahi, Alireza, et al.
Published: (2024)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
by: You, Haoran, et al.
Published: (2024)

How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression
by: Chen, Xingwu, et al.
Published: (2024)

Multi-Head Low-Rank Attention
by: Liu, Songtao, et al.
Published: (2026)

Benign Overfitting in Single-Head Attention
by: Magen, Roey, et al.
Published: (2024)

FedLLM: A Privacy-Preserving Federated Large Language Model for Explainable Traffic Flow Prediction
by: Kaur, Seerat, et al.
Published: (2026)

Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking
by: Shaj, Vaisakh, et al.
Published: (2026)

Parallax: Parameterized Local Linear Attention for Language Modeling
by: Zuo, Yifei, et al.
Published: (2026)

Can Large Language Models Transform Computational Social Science?
by: Ziems, Caleb, et al.
Published: (2023)

LoLCATs: On Low-Rank Linearizing of Large Language Models
by: Zhang, Michael, et al.
Published: (2024)

Memorization Capacity of Multi-Head Attention in Transformers
by: Mahdavi, Sadegh, et al.
Published: (2023)

The Effect of Attention Head Count on Transformer Approximation
by: Yu, Penghao, et al.
Published: (2025)

On the Origins of Linear Representations in Large Language Models
by: Jiang, Yibo, et al.
Published: (2024)

Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models
by: Zheng, Ziwei, et al.
Published: (2025)

DEI: Diversity in Evolutionary Inference for Quality-Diversity Search
by: Donaghy, John, et al.
Published: (2026)

FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
by: Smith, James Seale, et al.
Published: (2025)

Large Language Models aren't all that you need
by: Holla, Kiran Voderhobli, et al.
Published: (2024)

Log-Linear Attention
by: Guo, Han, et al.
Published: (2025)