Image de couverture de livre

Preconditioned Attention: Enhancing Efficiency in Transformers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Saratchandran, Hemanth
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Spectral Conditioning of Attention Improves Transformer Performance
par: Saratchandran, Hemanth, et autres
Publié: (2026)

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
par: Saratchandran, Hemanth, et autres
Publié: (2024)

Leaner Transformers: More Heads, Less Depth
par: Saratchandran, Hemanth, et autres
Publié: (2025)

Always Skip Attention
par: Ji, Yiping, et autres
Publié: (2025)

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy
par: Zheng, Jianqiao, et autres
Publié: (2026)

The Quantization Benefits of Residual-Free Transformers
par: Ji, Yiping, et autres
Publié: (2026)

Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
par: Saratchandran, Hemanth, et autres
Publié: (2024)

Architectural Strategies for the optimization of Physics-Informed Neural Networks
par: Saratchandran, Hemanth, et autres
Publié: (2024)

From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
par: Saratchandran, Hemanth, et autres
Publié: (2024)

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
par: Shinnick, Zachary, et autres
Publié: (2025)

Preconditioners for the Stochastic Training of Neural Fields
par: Chng, Shin-Fang, et autres
Publié: (2024)

Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
par: Garg, Arpit, et autres
Publié: (2025)

A Sampling Theory Perspective on Activations for Implicit Neural Representations
par: Saratchandran, Hemanth, et autres
Publié: (2024)

Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
par: Yao, Jiaqi, et autres
Publié: (2025)

Enhancing Transformers Through Conditioned Embedded Tokens
par: Saratchandran, Hemanth, et autres
Publié: (2025)

Cutting the Skip: Training Residual-Free Transformers
par: Ji, Yiping, et autres
Publié: (2025)

SineLoRA$Δ$: Sine-Activated Delta Compression
par: Gordon, Cameron, et autres
Publié: (2025)

From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
par: Zheng, Jianqiao, et autres
Publié: (2025)

D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations
par: Gordon, Cameron, et autres
Publié: (2024)

LUCID: Attention with Preconditioned Representations
par: Duvvuri, Sai Surya, et autres
Publié: (2026)

Procedural Pretraining: Warming Up Language Models with Abstract Data
par: Jiang, Liangze, et autres
Publié: (2026)

Efficient Learning With Sine-Activated Low-rank Matrices
par: Ji, Yiping, et autres
Publié: (2024)

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
par: Albert, Paul, et autres
Publié: (2025)

Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
par: Goel, Gautam, et autres
Publié: (2026)

Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
par: Ye, Zhenzhang, et autres
Publié: (2024)

Graph External Attention Enhanced Transformer
par: Liang, Jianqing, et autres
Publié: (2024)

Neuromorphic Mimicry Attacks Exploiting Brain-Inspired Computing for Covert Cyber Intrusions
par: Ravipati, Hemanth
Publié: (2025)

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
par: Zhou, Chenyu, et autres
Publié: (2026)

Enhancing Transformer Training Efficiency with Dynamic Dropout
par: Yan, Hanrui, et autres
Publié: (2024)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
par: Wen, Kaiyue, et autres
Publié: (2024)

Universal Sequence Preconditioning
par: Marsden, Annie, et autres
Publié: (2025)

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
par: Oh, Sehyeon, et autres
Publié: (2026)

A Ruelle dynamical zeta function for equivariant flows
par: Hochs, Peter, et autres
Publié: (2023)

An equivariant Guillemin trace formula
par: Hochs, Peter, et autres
Publié: (2025)

Enhancing Training Efficiency Using Packing with Flash Attention
par: Kundu, Achintya, et autres
Publié: (2024)

Fed-Meta-Align: A Similarity-Aware Aggregation and Personalization Pipeline for Federated TinyML on Heterogeneous Data
par: Macharla, Hemanth, et autres
Publié: (2025)

Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNER
par: Chandravamsi, Hemanth, et autres
Publié: (2025)

Advancing Household Robotics: Deep Interactive Reinforcement Learning for Efficient Training and Enhanced Performance
par: Soni, Arpita, et autres
Publié: (2024)

Enhancing Time Series Forecasting with Fuzzy Attention-Integrated Transformers
par: Chakraborty, Sanjay, et autres
Publié: (2025)

Hierarchical Transformer Preconditioning for Interactive Physics Simulation
par: Osborne, Carl, et autres
Publié: (2026)