Imagen de Portada

Beyond Linearity in Attention Projections: The Case for Nonlinear Queries

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autor principal:	Karbevski, Marko
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
por: Karbevski, Marko, et al.
Publicado: (2025)

Can an MLP Absorb Its Own Skip Connection?
por: Mijoski, Antonij, et al.
Publicado: (2026)

Exploring Local Explanations of Nonlinear Models Using Animated Linear Projections
por: Spyrison, Nicholas, et al.
Publicado: (2022)

Exact Linear Attention
por: Ou, Weinuo
Publicado: (2026)

Kaczmarz Linear Attention
por: Zou, Jiaxuan, et al.
Publicado: (2026)

Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
por: Zuo, Yifei, et al.
Publicado: (2025)

Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention
por: Hu, Wenjie, et al.
Publicado: (2025)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
por: Liang, Yingyu, et al.
Publicado: (2024)

On the Identifiability of Nonlinear ICA: Sparsity and Beyond
por: Zheng, Yujia, et al.
Publicado: (2022)

XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation
por: Kimura, Daichi, et al.
Publicado: (2025)

Tree-Sliced Wasserstein Distance with Nonlinear Projection
por: Tran, Thanh, et al.
Publicado: (2025)

Enhancing Linear Attention with Residual Learning
por: Lai, Xunhao, et al.
Publicado: (2025)

Surprised by Attention: Predictable Query Dynamics for Time Series Anomaly Detection
por: Özer, Kadir-Kaan, et al.
Publicado: (2026)

The Laplacian Keyboard: Beyond the Linear Span
por: Chandrasekar, Siddarth, et al.
Publicado: (2026)

Adaptive Memory Decay for Log-Linear Attention
por: Amin, Yaxita, et al.
Publicado: (2026)

KVBuffer: IO-aware Serving for Linear Attention
por: Zou, Longwei, et al.
Publicado: (2026)

State Rank Dynamics in Linear Attention LLMs
por: Sun, Ao, et al.
Publicado: (2026)

Linear Attention is Enough in Spatial-Temporal Forecasting
por: Ning, Xinyu
Publicado: (2024)

Linear Attention for Efficient Bidirectional Sequence Modeling
por: Afzal, Arshia, et al.
Publicado: (2025)

Conservative Contextual Bandits: Beyond Linear Representations
por: Deb, Rohan, et al.
Publicado: (2024)

Linear Projections of Teacher Embeddings for Few-Class Distillation
por: Loo, Noel, et al.
Publicado: (2024)

GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
por: Meng, Fanxu
Publicado: (2026)

Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
por: Beck, Maximilian, et al.
Publicado: (2025)

Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
por: Zhao, Weijie, et al.
Publicado: (2026)

ZeroS: Zero-Sum Linear Attention for Efficient Transformers
por: Lu, Jiecheng, et al.
Publicado: (2026)

Superiority of Multi-Head Attention in In-Context Linear Regression
por: Cui, Yingqian, et al.
Publicado: (2024)

RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts
por: Joshi, Sahil, et al.
Publicado: (2025)

LinearizeLLM: An Agent-Based Framework for LLM-Driven Exact Linear Reformulation of Nonlinear Optimization Problems
por: Kandora, Paul-Niklas Ken, et al.
Publicado: (2025)

Multi-Item-Query Attention for Stable Sequential Recommendation
por: Xu, Mingshi, et al.
Publicado: (2025)

From Projection to Prediction: Beyond Logits for Scalable Language Models
por: Dong, Jianbing, et al.
Publicado: (2025)

Beyond Similarity: Temporal Operator Attention for Time Series Analysis
por: Twitty, Jevon, et al.
Publicado: (2026)

Beyond Additivity: Sparse Isotonic Shapley Regression toward Nonlinear Explainability
por: She, Jialai
Publicado: (2025)

Higher-order Linear Attention
por: Zhang, Yifan, et al.
Publicado: (2025)

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain
por: Wi, Hyowon, et al.
Publicado: (2025)

MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
por: Chou, Yuhong, et al.
Publicado: (2024)

LNUCB-TA: Linear-nonlinear Hybrid Bandit Learning with Temporal Attention
por: Khosravi, Hamed, et al.
Publicado: (2025)

Cost-Optimal Grouped-Query Attention for Long-Context Modeling
por: Chen, Yingfa, et al.
Publicado: (2025)

Beyond Weighted Summation: Learnable Nonlinear Aggregation Functions for Robust Artificial Neurons
por: Bozyigit, Berke Deniz
Publicado: (2026)

Learning Linear Utility Functions From Pairwise Comparison Queries
por: Ge, Luise, et al.
Publicado: (2024)

NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inference
por: Yu, Jiangyong, et al.
Publicado: (2026)