Copertina

Surgical Repair of Collapsed Attention Heads in ALiBi Transformers

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autore principale:	Schallon, Palmer
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
di: Pawlowsky, Moritz, et al.
Pubblicazione: (2026)

Multi-modal, multi-scale representation learning for satellite imagery analysis just needs a good ALiBi
di: Kage, Patrick, et al.
Pubblicazione: (2026)

Interpreting Transformers Through Attention Head Intervention
di: Kadem, Mason, et al.
Pubblicazione: (2026)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
di: Fu, Zizhuo, et al.
Pubblicazione: (2026)

Improving Transformers with Dynamically Composable Multi-Head Attention
di: Xiao, Da, et al.
Pubblicazione: (2024)

BiMind: A Dual-Head Reasoning Model with Attention-Geometry Adapter for Incorrect Information Detection
di: Zhang, Zhongxing, et al.
Pubblicazione: (2026)

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
di: Csordás, Róbert, et al.
Pubblicazione: (2023)

Knocking-Heads Attention
di: Zhou, Zhanchao, et al.
Pubblicazione: (2025)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
di: Musat, Tiberiu
Pubblicazione: (2024)

Head Pursuit: Probing Attention Specialization in Multimodal Transformers
di: Basile, Lorenzo, et al.
Pubblicazione: (2025)

Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
di: Wang, Junxuan, et al.
Pubblicazione: (2025)

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
di: Cirrincione, Giansalvo
Pubblicazione: (2026)

Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads
di: Yang, Yi, et al.
Pubblicazione: (2023)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
di: Balogh, Peter
Pubblicazione: (2026)

RecurFormer: Not All Transformer Heads Need Self-Attention
di: Yan, Ruiqing, et al.
Pubblicazione: (2024)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
di: Chen, Yilong, et al.
Pubblicazione: (2024)

S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
di: Lin, Xihui, et al.
Pubblicazione: (2024)

Inferring Functionality of Attention Heads from their Parameters
di: Elhelo, Amit, et al.
Pubblicazione: (2024)

Head-wise Shareable Attention for Large Language Models
di: Cao, Zouying, et al.
Pubblicazione: (2024)

Attention Heads of Large Language Models: A Survey
di: Zheng, Zifan, et al.
Pubblicazione: (2024)

AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
di: Hua, Kai, et al.
Pubblicazione: (2025)

Sycophancy Hides Linearly in the Attention Heads
di: Genadi, Rifo, et al.
Pubblicazione: (2026)

ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
di: Shao, Jintian, et al.
Pubblicazione: (2025)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
di: Lu, Yi, et al.
Pubblicazione: (2024)

MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads
di: Liu, Weihao, et al.
Pubblicazione: (2025)

Efficient Machine Translation with a BiLSTM-Attention Approach
di: Wu, Yuxu, et al.
Pubblicazione: (2024)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
di: Ji, Tao, et al.
Pubblicazione: (2025)

Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs
di: Vazhentsev, Artem, et al.
Pubblicazione: (2025)

Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads
di: Yeo, Wei Jie, et al.
Pubblicazione: (2025)

Debiasing LLMs by Masking Unfairness-Driving Attention Heads
di: Han, Tingxu, et al.
Pubblicazione: (2025)

Analyzing Multi-Head Attention on Trojan BERT Models
di: Wang, Jingwei
Pubblicazione: (2024)

CHAI: Clustered Head Attention for Efficient LLM Inference
di: Agarwal, Saurabh, et al.
Pubblicazione: (2024)

Latent Multi-Head Attention for Small Language Models
di: Mehta, Sushant, et al.
Pubblicazione: (2025)

A Transformer with Stack Attention
di: Li, Jiaoda, et al.
Pubblicazione: (2024)

Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
di: Yamagiwa, Hiroaki, et al.
Pubblicazione: (2026)

ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
di: Liu, Zhuorui, et al.
Pubblicazione: (2025)

AIR: Post-training Data Selection for Reasoning via Attention Head Influence
di: Liu, Jinrui, et al.
Pubblicazione: (2025)

MossNet: Mixture of State-Space Experts is a Multi-Head Attention
di: Tuli, Shikhar, et al.
Pubblicazione: (2025)

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
di: Xiao, Guangxuan, et al.
Pubblicazione: (2024)

Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR
di: Ratajczak, Martin, et al.
Pubblicazione: (2025)