Image de couverture de livre

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Chen, Siyu, Sheen, Heejune, Wang, Tianhao, Yang, Zhuoran
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence Computation and Language Optimization and Control
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
par: Chen, Siyu, et autres
Publié: (2024)

Implicit Regularization of Gradient Flow on One-Layer Softmax Attention
par: Sheen, Heejune, et autres
Publié: (2024)

Taming Polysemanticity in LLMs: Provable Feature Recovery via Sparse Autoencoders
par: Chen, Siyu, et autres
Publié: (2025)

When and How Unlabeled Data Provably Improve In-Context Learning
par: Li, Yingcong, et autres
Publié: (2025)

Agentic Transformers Provably Learn to Search via Reinforcement Learning
par: Yang, Tong, et autres
Publié: (2026)

Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization
par: Huang, Yu, et autres
Publié: (2025)

Transformers as Support Vector Machines
par: Tarzanagh, Davoud Ataee, et autres
Publié: (2023)

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
par: Li, Zihao, et autres
Publié: (2024)

Leveraging Large Language Models for Solving Rare MIP Challenges
par: Wang, Teng, et autres
Publié: (2024)

Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup
par: Chidambaram, Muthu, et autres
Publié: (2022)

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
par: He, Jianliang, et autres
Publié: (2026)

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
par: Fernando, Heshan, et autres
Publié: (2024)

Solving General Natural-Language-Description Optimization Problems with Large Language Models
par: Zhang, Jihai, et autres
Publié: (2024)

How Well Can Transformers Emulate In-context Newton's Method?
par: Giannou, Angeliki, et autres
Publié: (2024)

Variational Learning is Effective for Large Deep Networks
par: Shen, Yuesong, et autres
Publié: (2024)

Reinforcement Learning from Human Feedback with Active Queries
par: Ji, Kaixuan, et autres
Publié: (2024)

Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
par: Li, Yingcong, et autres
Publié: (2025)

DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
par: Gurses, Selcuk, et autres
Publié: (2025)

ACING: Actor-Critic for Instruction Learning in Black-Box LLMs
par: Kharrat, Salma, et autres
Publié: (2024)

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks
par: Liu, Xin, et autres
Publié: (2022)

Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent
par: Yang, Tong, et autres
Publié: (2025)

Absorber LLM: Harnessing Causal Synchronization for Test-Time Training
par: Zhang, Zhixin, et autres
Publié: (2026)

One-Shot Safety Alignment for Large Language Models via Optimal Dualization
par: Huang, Xinmeng, et autres
Publié: (2024)

Causal LLM Routing: End-to-End Regret Minimization from Observational Data
par: Tsiourvas, Asterios, et autres
Publié: (2025)

Reward Collapse in Aligning Large Language Models
par: Song, Ziang, et autres
Publié: (2023)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
par: Gautam, Tanmay, et autres
Publié: (2024)

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
par: Pan, Rui, et autres
Publié: (2024)

Mechanics of Next Token Prediction with Self-Attention
par: Li, Yingcong, et autres
Publié: (2024)

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
par: Tao, Hongyi, et autres
Publié: (2026)

Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
par: Li, Yingcong, et autres
Publié: (2024)

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence
par: Xiao, Minheng, et autres
Publié: (2024)

From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems
par: He, Jianliang, et autres
Publié: (2024)

Exploring the Robustness of In-Context Learning with Noisy Labels
par: Cheng, Chen, et autres
Publié: (2024)

Provable Offline Reinforcement Learning for Structured Cyclic MDPs
par: Lee, Kyungbok, et autres
Publié: (2026)

ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertise
par: Guo, Xingang, et autres
Publié: (2024)

Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
par: Zhang, Zhixin, et autres
Publié: (2025)

Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency
par: Cai, Qi, et autres
Publié: (2022)

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
par: Wang, Jinbo, et autres
Publié: (2025)

Training Infinitely Deep and Wide Transformers
par: Barboni, Raphaël, et autres
Publié: (2026)

Provable Acceleration for Diffusion Models under Minimal Assumptions
par: Li, Gen, et autres
Publié: (2024)