Image de couverture de livre

The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Edelman, Benjamin L., Edelman, Ezra, Goel, Surbhi, Malach, Eran, Tsilivis, Nikolaos
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Reliable Abstention under Adversarial Injections: Tight Lower Bounds and New Upper Bounds
par: Edelman, Ezra, et autres
Publié: (2026)

How Reinforcement Learning After Next-Token Prediction Facilitates Learning
par: Tsilivis, Nikolaos, et autres
Publié: (2025)

Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Ones
par: Mirtaheri, Parsa, et autres
Publié: (2025)

Auto-Regressive Next-Token Predictors are Universal Learners
par: Malach, Eran
Publié: (2023)

Transcendence: Generative Models Can Outperform The Experts That Train Them
par: Zhang, Edwin, et autres
Publié: (2024)

The Power of Random Features and the Limits of Distribution-Free Gradient Descent
par: Karchmer, Ari, et autres
Publié: (2025)

Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts
par: Wang, James, et autres
Publié: (2026)

Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift
par: Goel, Surbhi, et autres
Publié: (2026)

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
par: Tsilivis, Nikolaos, et autres
Publié: (2024)

The Price of Implicit Bias in Adversarially Robust Generalization
par: Tsilivis, Nikolaos, et autres
Publié: (2024)

Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints
par: Pal, Avik, et autres
Publié: (2025)

LLM Priors for ERM over Programs
par: Singhal, Shivam, et autres
Publié: (2025)

Matrix Calculus (for Machine Learning and Beyond)
par: Bright, Paige, et autres
Publié: (2025)

Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations
par: Qiu, GuanWen, et autres
Publié: (2024)

On the Robustness of Neural Collapse and the Neural Collapse of Robustness
par: Su, Jingtong, et autres
Publié: (2023)

To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
par: Qin, Tian, et autres
Publié: (2025)

Why Do Transformers Fail to Forecast Time Series In-Context?
par: Zhou, Yufa, et autres
Publié: (2025)

Markov Chain Estimation with In-Context Learning
par: Lepage, Simon, et autres
Publié: (2025)

Learning to Think from Multiple Thinkers
par: Joshi, Nirmit, et autres
Publié: (2026)

Universal Length Generalization with Turing Programs
par: Hou, Kaiying, et autres
Publié: (2024)

A Theory of Learning with Autoregressive Chain of Thought
par: Joshi, Nirmit, et autres
Publié: (2025)

What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
par: Ekbote, Chanakya, et autres
Publié: (2025)

Distinguishing the Knowable from the Unknowable with Language Models
par: Ahdritz, Gustaf, et autres
Publié: (2024)

On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds
par: Vilucchio, Matteo, et autres
Publié: (2024)

Tolerant Algorithms for Learning with Arbitrary Covariate Shift
par: Goel, Surbhi, et autres
Publié: (2024)

Repeat After Me: Transformers are Better than State Space Models at Copying
par: Jelassi, Samy, et autres
Publié: (2024)

Selective Induction Heads: How Transformers Select Causal Structures In Context
par: D'Angelo, Francesco, et autres
Publié: (2025)

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
par: Zhao, Rosie, et autres
Publié: (2025)

From Markov to Laplace: How Mamba In-Context Learns Markov Chains
par: Bondaschi, Marco, et autres
Publié: (2025)

What are human values, and how do we align AI to them?
par: Klingefjord, Oliver, et autres
Publié: (2024)

Don't Stop Me Now: Embedding Based Scheduling for LLMs
par: Shahout, Rana, et autres
Publié: (2024)

Testing Noise Assumptions of Learning Algorithms
par: Goel, Surbhi, et autres
Publié: (2025)

N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs
par: Zisman, Ilya, et autres
Publié: (2024)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
par: Brandfonbrener, David, et autres
Publié: (2024)

Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
par: Zhang, Hanlin, et autres
Publié: (2023)

Feature emergence via margin maximization: case studies in algebraic tasks
par: Morwani, Depen, et autres
Publié: (2023)

PlaceNav: Topological Navigation through Place Recognition
par: Suomela, Lauri, et autres
Publié: (2023)

A New Perspective on Shampoo's Preconditioner
par: Morwani, Depen, et autres
Publié: (2024)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
par: Prabhakar, Akshara, et autres
Publié: (2024)

Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks
par: Sabbaghi, Mahdi, et autres
Publié: (2024)