Imagen de Portada

How do Transformers perform In-Context Autoregressive Learning?

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Sander, Michael E., Giryes, Raja, Suzuki, Taiji, Blondel, Mathieu, Peyré, Gabriel
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Towards Understanding the Universality of Transformers for Next-Token Prediction
por: Sander, Michael E., et al.
Publicado: (2024)

Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
por: Kim, Juno, et al.
Publicado: (2024)

Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
por: Blondel, Mathieu, et al.
Publicado: (2025)

Joint Learning of Energy-based Models and their Partition Function
por: Sander, Michael E., et al.
Publicado: (2025)

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
por: Wakayama, Tomoya, et al.
Publicado: (2025)

Transformers are Minimax Optimal Nonparametric In-Context Learners
por: Kim, Juno, et al.
Publicado: (2024)

Differentiable Knapsack and Top-k Operators via Dynamic Programming
por: Vivier-Ardisson, Germain, et al.
Publicado: (2026)

Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
por: Oh, Junsoo, et al.
Publicado: (2025)

Transformers Provably Solve Parity Efficiently with Chain of Thought
por: Kim, Juno, et al.
Publicado: (2024)

State Space Models are Provably Comparable to Transformers in Dynamic Token Selection
por: Nishikawa, Naoki, et al.
Publicado: (2024)

Transformers as Measure-Theoretic Associative Memory: A Statistical Perspective and Minimax Optimality
por: Kawata, Ryotaro, et al.
Publicado: (2026)

Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input
por: Takakura, Shokichi, et al.
Publicado: (2023)

Loss Functions and Operators Generated by f-Divergences
por: Roulet, Vincent, et al.
Publicado: (2025)

Optimal and Diffusion Transports in Machine Learning
por: Peyré, Gabriel
Publicado: (2025)

Transformative or Conservative? Conservation laws for ResNets and Transformers
por: Marcotte, Sibylle, et al.
Publicado: (2025)

How Smooth Is Attention?
por: Castin, Valérie, et al.
Publicado: (2023)

ZOQO: Zero-Order Quantized Optimization
por: Bar, Noga, et al.
Publicado: (2025)

Robust Sublinear Convergence Rates for Iterative Bregman Projections
por: Peyré, Gabriel
Publicado: (2026)

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning
por: Bu, Dake, et al.
Publicado: (2024)

Pruning at Initialization -- A Sketching Perspective
por: Bar, Noga, et al.
Publicado: (2023)

Muon Dynamics as a Spectral Wasserstein Flow
por: Peyré, Gabriel
Publicado: (2026)

Optimal Transport for Machine Learners
por: Peyré, Gabriel
Publicado: (2025)

DINOv2 based Self Supervised Learning For Few Shot Medical Image Segmentation
por: Ayzenberg, Lev, et al.
Publicado: (2024)

Diverse Subset Selection via Norm-Based Sampling and Orthogonality
por: Bar, Noga, et al.
Publicado: (2024)

Learning with Local Search MCMC Layers
por: Vivier-Ardisson, Germain, et al.
Publicado: (2025)

Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective
por: Takakura, Shokichi, et al.
Publicado: (2024)

Deep Two-Way Matrix Reordering for Relational Data Analysis
por: Watanabe, Chihiro, et al.
Publicado: (2021)

The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge
por: Awano, Ryoya, et al.
Publicado: (2026)

AutoLL: Automatic Linear Layout of Graphs based on Deep Neural Network
por: Watanabe, Chihiro, et al.
Publicado: (2021)

Test time training enhances in-context learning of nonlinear functions
por: Kuwataka, Kento, et al.
Publicado: (2025)

Multiplicative Reweighting for Robust Neural Network Optimization
por: Bar, Noga, et al.
Publicado: (2021)

The Elements of Differentiable Programming
por: Blondel, Mathieu, et al.
Publicado: (2024)

From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
por: Kawata, Ryotaro, et al.
Publicado: (2025)

Transformers are Universal In-context Learners
por: Furuya, Takashi, et al.
Publicado: (2024)

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
por: Higuchi, Rei, et al.
Publicado: (2025)

Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
por: Jiang, Jiarui, et al.
Publicado: (2025)

CARES: Context-Aware Resolution Selector for VLMs
por: Kimhi, Moshe, et al.
Publicado: (2025)

UDPM: Upsampling Diffusion Probabilistic Models
por: Abu-Hussein, Shady, et al.
Publicado: (2023)

How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
por: Higuchi, Rei, et al.
Publicado: (2026)

Clustering in Deep Stochastic Transformers
por: Fedorov, Lev, et al.
Publicado: (2026)