Cover Image

Linear Transformers with Learnable Kernel Functions are Better In-Context Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Aksenov, Yaroslav, Balagansky, Nikita, Vaina, Sofia Maria Lo Cicero, Shaposhnikov, Boris, Gorbatovski, Alexey, Gavrilov, Daniil
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Diffusion Language Models Generation Can Be Halted Early
by: Vaina, Sofia Maria Lo Cicero, et al.
Published: (2023)

Learn Your Reference Model for Real Good Alignment
by: Gorbatovski, Alexey, et al.
Published: (2024)

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
by: Laptev, Daniil, et al.
Published: (2025)

You Do Not Fully Utilize Transformer's Representation Capacity
by: Gerasimov, Gleb, et al.
Published: (2025)

Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
by: Kurochkin, Vadim, et al.
Published: (2025)

Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors
by: Sinii, Viacheslav, et al.
Published: (2025)

Teach Old SAEs New Domain Tricks with Boosting
by: Koriagin, Nikita, et al.
Published: (2025)

Steering LLM Reasoning Through Bias-Only Adaptation
by: Sinii, Viacheslav, et al.
Published: (2025)

Trust-Region Behavior Blending for On-Policy Distillation
by: Plyusov, Daniil, et al.
Published: (2026)

The Differences Between Direct Alignment Algorithms are a Blur
by: Gorbatovski, Alexey, et al.
Published: (2025)

Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy
by: Balagansky, Nikita, et al.
Published: (2025)

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
by: Plyusov, Daniil, et al.
Published: (2026)

Mechanistic Permutability: Match Features Across Layers
by: Balagansky, Nikita, et al.
Published: (2024)

ESSA: Evolutionary Strategies for Scalable Alignment
by: Korotyshova, Daria, et al.
Published: (2025)

Next Embedding Prediction Makes World Models Stronger
by: Bredis, George, et al.
Published: (2026)

Mashup Learning: Faster Finetuning by Remixing Past Checkpoints
by: Vaina, Sofia Maria Lo Cicero, et al.
Published: (2026)

Guided Star-Shaped Masked Diffusion
by: Meshchaninov, Viacheslav, et al.
Published: (2025)

Reinforcement learning for question answering in programming domain using public community scoring as a human feedback
by: Gorbatovski, Alexey, et al.
Published: (2024)

Learnable Permutation for Structured Sparsity on Transformer Models
by: Li, Zekai, et al.
Published: (2026)

Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
by: Soni, Nikita, et al.
Published: (2024)

Transformer Based Linear Attention with Optimized GPU Kernel Implementation
by: Gerami, Armin, et al.
Published: (2025)

LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs
by: Gordon, Ofir, et al.
Published: (2026)

Alt-Text with Context: Improving Accessibility for Images on Twitter
by: Srivatsan, Nikita, et al.
Published: (2023)

VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks
by: Diatlova, Daria, et al.
Published: (2025)

On the Robustness of Transformers against Context Hijacking for Linear Classification
by: Li, Tianle, et al.
Published: (2025)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
by: He, Mutian, et al.
Published: (2025)

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
by: Liu, Shang, et al.
Published: (2024)

Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention
by: Kiruluta, Andrew, et al.
Published: (2025)

On the Learnability of Watermarks for Language Models
by: Gu, Chenchen, et al.
Published: (2023)

When an LLM is apprehensive about its answers -- and when its uncertainty is justified
by: Sychev, Petr, et al.
Published: (2025)

Complexity-aware fine-tuning
by: Goncharov, Andrey, et al.
Published: (2025)

Core Context Aware Transformers for Long Context Language Modeling
by: Chen, Yaofo, et al.
Published: (2024)

DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning
by: Ignatev, Daniil, et al.
Published: (2025)

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
by: Sundaram, Shobhita, et al.
Published: (2026)

Quantization of Large Language Models with an Overdetermined Basis
by: Merkulov, Daniil, et al.
Published: (2024)

Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention
by: Kiruluta, Andrew
Published: (2025)

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
by: Fu, Deqing, et al.
Published: (2023)

In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization
by: Zhang, Ruiqi, et al.
Published: (2024)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
by: Collins, Liam, et al.
Published: (2024)

Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models
by: Browder, Michael, et al.
Published: (2026)