Cover Image

Infinite Limits of Multi-head Transformer Dynamics

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bordelon, Blake, Chaudhry, Hamza Tahir, Pehlevan, Cengiz
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Disordered Systems and Neural Networks
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning
by: Bordelon, Blake, et al.
Published: (2026)

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
by: Bordelon, Blake, et al.
Published: (2025)

Transfer Learning in Infinite Width Feature Learning Networks
by: Lauditi, Clarissa, et al.
Published: (2025)

A Dynamical Model of Neural Scaling Laws
by: Bordelon, Blake, et al.
Published: (2024)

How Feature Learning Can Improve Neural Scaling Laws
by: Bordelon, Blake, et al.
Published: (2024)

Adaptive kernel predictors from feature-learning infinite limits of neural networks
by: Lauditi, Clarissa, et al.
Published: (2025)

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
by: Lauditi, Clarissa, et al.
Published: (2026)

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
by: Bordelon, Blake, et al.
Published: (2025)

Grokking as the Transition from Lazy to Rich Training Dynamics
by: Kumar, Tanishq, et al.
Published: (2023)

No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions
by: Ruben, Benjamin S., et al.
Published: (2024)

Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models
by: Atanasov, Alexander, et al.
Published: (2025)

Dynamically Learning to Integrate in Recurrent Neural Networks
by: Bordelon, Blake, et al.
Published: (2025)

Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model
by: Bordelon, Blake, et al.
Published: (2026)

Nadaraya-Watson kernel smoothing as a random energy model
by: Zavatone-Veth, Jacob A., et al.
Published: (2024)

A solvable model of learning generative diffusion: theory and insights
by: Cui, Hugo, et al.
Published: (2025)

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles
by: Ruben, Benjamin S., et al.
Published: (2023)

Risk and cross validation in ridge regression with correlated samples
by: Atanasov, Alexander, et al.
Published: (2024)

Scaling and renormalization in high-dimensional regression
by: Atanasov, Alexander, et al.
Published: (2024)

How does training shape the Riemannian geometry of neural network representations?
by: Zavatone-Veth, Jacob A., et al.
Published: (2023)

Asymptotic theory of in-context learning by linear attention
by: Lu, Yue M., et al.
Published: (2024)

A note on the dynamics of extended-context disordered kinetic spin models
by: Zavatone-Veth, Jacob A., et al.
Published: (2025)

Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers
by: Davidovich, Orit, et al.
Published: (2026)

Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limit
by: Meng, Lineghuan, et al.
Published: (2024)

Geometric Dynamics of Signal Propagation Predict Trainability of Transformers
by: Cowsik, Aditya, et al.
Published: (2024)

Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers
by: Mainali, Nischal, et al.
Published: (2025)

High-Dimensional Limit of Stochastic Gradient Flow via Dynamical Mean-Field Theory
by: Nishiyama, Sota, et al.
Published: (2026)

Specialization of softmax attention heads: insights from the high-dimensional single-location model
by: Sagitova, M., et al.
Published: (2026)

Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning
by: Ariosto, Sebastiano
Published: (2025)

Towards Understanding Inductive Bias in Transformers: A View From Infinity
by: Lavie, Itay, et al.
Published: (2024)

Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory
by: Nishiyama, Sota, et al.
Published: (2025)

Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
by: Staats, Max, et al.
Published: (2024)

Dynamical Regimes of Multimodal Diffusion Models
by: Albrychiewicz, Emil, et al.
Published: (2026)

Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
by: Cagnetta, Francesco, et al.
Published: (2025)

Dynamics of Meta-learning Representation in the Teacher-student Scenario
by: Wang, Hui, et al.
Published: (2024)

Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
by: Jain, Anchit, et al.
Published: (2024)

Dynamical Mean-Field Theory of Self-Attention Neural Networks
by: Poc-López, Ángel, et al.
Published: (2024)

The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
by: Patel, Nishil, et al.
Published: (2023)

The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models
by: Nicoletti, Flavio, et al.
Published: (2026)

Growing Neural Networks: Dynamic Evolution through Gradient Descent
by: Radhakrishnan, Anil, et al.
Published: (2025)

Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks
by: Montanari, Andrea, et al.
Published: (2025)