Cover Image

Limitations of Normalization in Attention Mechanism

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mudarisov, Timur, Burtsev, Mikhail, Petrova, Tatiana, State, Radu
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Geometric Analysis of Token Selection in Multi-Head Attention
by: Mudarisov, Timur, et al.
Published: (2026)

Wikontic: Constructing Wikidata-Aligned, Ontology-Aware Knowledge Graphs with Large Language Models
by: Chepurova, Alla, et al.
Published: (2025)

Scaling Transformer to 1M tokens and beyond with RMT
by: Bulatov, Aydar, et al.
Published: (2023)

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
by: Kuratov, Yuri, et al.
Published: (2024)

Associative Recurrent Memory Transformer
by: Rodkin, Ivan, et al.
Published: (2024)

Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
by: Kuratov, Yuri, et al.
Published: (2025)

LongKey: Keyphrase Extraction for Long Documents
by: Alves, Jeovane Honorio, et al.
Published: (2024)

Learning Elementary Cellular Automata with Transformers
by: Burtsev, Mikhail
Published: (2024)

Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
by: Ok, Hyunjong, et al.
Published: (2026)

Scaling Bidirectional Spans and Span Violations in Attention Mechanism
by: Kim, Jongwook, et al.
Published: (2025)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
by: Zhang, Stephen, et al.
Published: (2025)

A Simple Attention-Based Mechanism for Bimodal Emotion Classification
by: Elabd, Mazen, et al.
Published: (2024)

Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining
by: Huang, Ruizhe, et al.
Published: (2025)

Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures
by: Thiombiano, Abdoul Majid O., et al.
Published: (2025)

Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
by: Leemann, Tobias, et al.
Published: (2024)

UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions
by: Rogoz, Ana-Cristina, et al.
Published: (2024)

On the Volatility of Shapley-Based Contribution Metrics in Federated Learning
by: Geimer, Arno, et al.
Published: (2024)

Detecting Suicidal Ideation in Text with Interpretable Deep Learning: A CNN-BiGRU with Attention Mechanism
by: Bhuiyan, Mohaiminul Islam, et al.
Published: (2025)

CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning
by: Dragomir, Alexandra, et al.
Published: (2026)

A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus
by: Poesina, Eduard, et al.
Published: (2024)

From Multi-Agent Systems and the Semantic Web to Agentic AI: A Unified Narrative of the Web of Agents
by: Petrova, Tatiana, et al.
Published: (2025)

Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
by: Jarca, Andrei, et al.
Published: (2025)

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
by: Kuratov, Yuri, et al.
Published: (2024)

Mechanics of Next Token Prediction with Self-Attention
by: Li, Yingcong, et al.
Published: (2024)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
by: Deng, Yichuan, et al.
Published: (2024)

DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
by: Huang, Yuxiang, et al.
Published: (2026)

PoPreRo: A New Dataset for Popularity Prediction of Romanian Reddit Posts
by: Rogoz, Ana-Cristina, et al.
Published: (2024)

Learning Using Generated Privileged Information by Text-to-Image Diffusion Models
by: Menadil, Rafael-Edy, et al.
Published: (2023)

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
by: Kim, Jeonghoon, et al.
Published: (2025)

Discourse-Aware In-Context Learning for Temporal Expression Normalization
by: Gautam, Akash Kumar, et al.
Published: (2024)

Attention Needs to Focus: A Unified Perspective on Attention Allocation
by: Fu, Zichuan, et al.
Published: (2026)

Efficiently Dispatching Flash Attention For Partially Filled Attention Masks
by: Sharma, Agniv, et al.
Published: (2024)

Incorporating Graph Attention Mechanism into Geometric Problem Solving Based on Deep Reinforcement Learning
by: Zhong, Xiuqin, et al.
Published: (2024)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
by: Saxena, Utkarsh, et al.
Published: (2024)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
by: Knupp, Jonas, et al.
Published: (2026)

Large Language Models in the Task of Automatic Validation of Text Classifier Predictions
by: Tsymbalov, Aleksandr, et al.
Published: (2025)

Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing
by: Mirtaheri, Parsa, et al.
Published: (2026)

On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions
by: Stollenwerk, Felix
Published: (2025)

Geometric Interpretation of Layer Normalization and a Comparative Analysis with RMSNorm
by: Gupta, Akshat, et al.
Published: (2024)