Imagen de Portada

Linear Transformers are Versatile In-Context Learners

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Vladymyrov, Max, von Oswald, Johannes, Sandler, Mark, Ge, Rong
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learning
por: Vladymyrov, Max, et al.
Publicado: (2023)

Contextually Guided Transformers via Low-Rank Adaptation
por: Zhmoginov, Andrey, et al.
Publicado: (2025)

Long Context In-Context Compression by Getting to the Gist of Gisting
por: Petrov, Aleksandar, et al.
Publicado: (2025)

Uncovering mesa-optimization algorithms in Transformers
por: von Oswald, Johannes, et al.
Publicado: (2023)

Narrowing the Focus: Learned Optimizers for Pretrained Models
por: Kristiansen, Gus, et al.
Publicado: (2024)

Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens
por: Anwar, Usman, et al.
Publicado: (2024)

Learning Randomized Algorithms with Transformers
por: von Oswald, Johannes, et al.
Publicado: (2024)

State Soup: In-Context Skill Learning, Retrieval and Mixing
por: Pióro, Maciej, et al.
Publicado: (2024)

Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
por: Zhmoginov, Andrey, et al.
Publicado: (2025)

Transformers are Minimax Optimal Nonparametric In-Context Learners
por: Kim, Juno, et al.
Publicado: (2024)

Rotary Masked Autoencoders are Versatile Learners
por: Zivanovic, Uros, et al.
Publicado: (2025)

Fine-Tuning Dynamics of In-Context Factual Recall in Transformers
por: Huang, Ruomin, et al.
Publicado: (2026)

Diffusion Language Models Are Versatile Protein Learners
por: Wang, Xinyou, et al.
Publicado: (2024)

Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs
por: Ling, Tianheng, et al.
Publicado: (2025)

Fast Differentiable Modal Simulation of Non-linear Strings, Membranes, and Plates
por: Diaz, Rodrigo, et al.
Publicado: (2025)

InAttention: Linear Context Scaling for Transformers
por: Eisner, Joseph
Publicado: (2024)

Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
por: Polubarov, Andrei, et al.
Publicado: (2026)

Plain Transformers Can be Powerful Graph Learners
por: Ma, Liheng, et al.
Publicado: (2025)

Dynamics of Transient Structure in In-Context Linear Regression Transformers
por: Carroll, Liam, et al.
Publicado: (2025)

Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners
por: Kumano, Soichiro, et al.
Publicado: (2025)

Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners
por: Araslanov, Nikita, et al.
Publicado: (2026)

Weight decay induces low-rank attention layers
por: Kobayashi, Seijin, et al.
Publicado: (2024)

PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations
por: Holzschuh, Benjamin, et al.
Publicado: (2025)

Mixtures of In-Context Learners
por: Hong, Giwon, et al.
Publicado: (2024)

Lifelong Learner: Discovering Versatile Neural Solvers for Vehicle Routing Problems
por: Feng, Shaodi, et al.
Publicado: (2025)

Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
por: Chen, Brian K, et al.
Publicado: (2024)

Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
por: Xie, Zixuan, et al.
Publicado: (2026)

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
por: Pandey, Vishal, et al.
Publicado: (2026)

Incentivizing Exploration with Linear Contexts and Combinatorial Actions
por: Sellke, Mark
Publicado: (2023)

Large (Vision) Language Models are Unsupervised In-Context Learners
por: Gadetsky, Artyom, et al.
Publicado: (2025)

LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
por: Hili, Youssef Attia El, et al.
Publicado: (2025)

Transformers are Universal In-context Learners
por: Furuya, Takashi, et al.
Publicado: (2024)

Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context
por: Cheng, Xiang, et al.
Publicado: (2023)

In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation
por: Cole, Frank, et al.
Publicado: (2025)

Learning and Unlearning of Fabricated Knowledge in Language Models
por: Sun, Chen, et al.
Publicado: (2024)

Non-Convex Optimization with Spectral Radius Regularization
por: Sandler, Adam, et al.
Publicado: (2021)

Linear Transformers with Learnable Kernel Functions are Better In-Context Models
por: Aksenov, Yaroslav, et al.
Publicado: (2024)

Fine-Tuned In-Context Learners for Efficient Adaptation
por: Bornschein, Jorg, et al.
Publicado: (2025)

Are Large Language Models In-Context Graph Learners?
por: Li, Jintang, et al.
Publicado: (2025)

From Growing to Looping: A Unified View of Iterative Computation in LLMs
por: Kapl, Ferdinand, et al.
Publicado: (2026)