Copertina

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Bordelon, Blake, Letey, Mary I., Pehlevan, Cengiz
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Disordered Systems and Neural Networks
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
di: Bordelon, Blake, et al.
Pubblicazione: (2025)

A Dynamical Model of Neural Scaling Laws
di: Bordelon, Blake, et al.
Pubblicazione: (2024)

Transfer Learning in Infinite Width Feature Learning Networks
di: Lauditi, Clarissa, et al.
Pubblicazione: (2025)

How Feature Learning Can Improve Neural Scaling Laws
di: Bordelon, Blake, et al.
Pubblicazione: (2024)

Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning
di: Bordelon, Blake, et al.
Pubblicazione: (2026)

Adaptive kernel predictors from feature-learning infinite limits of neural networks
di: Lauditi, Clarissa, et al.
Pubblicazione: (2025)

Infinite Limits of Multi-head Transformer Dynamics
di: Bordelon, Blake, et al.
Pubblicazione: (2024)

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
di: Lauditi, Clarissa, et al.
Pubblicazione: (2026)

Grokking as the Transition from Lazy to Rich Training Dynamics
di: Kumar, Tanishq, et al.
Pubblicazione: (2023)

Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model
di: Bordelon, Blake, et al.
Pubblicazione: (2026)

Asymptotic theory of in-context learning by linear attention
di: Lu, Yue M., et al.
Pubblicazione: (2024)

Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models
di: Atanasov, Alexander, et al.
Pubblicazione: (2025)

Dynamically Learning to Integrate in Recurrent Neural Networks
di: Bordelon, Blake, et al.
Pubblicazione: (2025)

No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions
di: Ruben, Benjamin S., et al.
Pubblicazione: (2024)

Scaling and renormalization in high-dimensional regression
di: Atanasov, Alexander, et al.
Pubblicazione: (2024)

Nadaraya-Watson kernel smoothing as a random energy model
di: Zavatone-Veth, Jacob A., et al.
Pubblicazione: (2024)

A solvable model of learning generative diffusion: theory and insights
di: Cui, Hugo, et al.
Pubblicazione: (2025)

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles
di: Ruben, Benjamin S., et al.
Pubblicazione: (2023)

Risk and cross validation in ridge regression with correlated samples
di: Atanasov, Alexander, et al.
Pubblicazione: (2024)

How does training shape the Riemannian geometry of neural network representations?
di: Zavatone-Veth, Jacob A., et al.
Pubblicazione: (2023)

Learning Linear Regression with Low-Rank Tasks in-Context
di: Takanami, Kaito, et al.
Pubblicazione: (2025)

A note on the dynamics of extended-context disordered kinetic spin models
di: Zavatone-Veth, Jacob A., et al.
Pubblicazione: (2025)

Explaining Neural Scaling Laws
di: Bahri, Yasaman, et al.
Pubblicazione: (2021)

Neural Scaling Laws Rooted in the Data Distribution
di: Brill, Ari
Pubblicazione: (2024)

Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning
di: Ariosto, Sebastiano
Pubblicazione: (2025)

Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
di: Cagnetta, Francesco, et al.
Pubblicazione: (2025)

Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers
di: Mainali, Nischal, et al.
Pubblicazione: (2025)

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
di: Sarmiento, Lucas Fernandez
Pubblicazione: (2026)

From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
di: Rubin, Noa, et al.
Pubblicazione: (2025)

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
di: Boncoraglio, Fabrizio, et al.
Pubblicazione: (2025)

Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression
di: Clarté, Lucas, et al.
Pubblicazione: (2024)

Statistical Advantage of Softmax Attention: Insights from Single-Location Regression
di: Duranthon, O., et al.
Pubblicazione: (2025)

Asymmetric Scaling Laws from Sparse Features
di: Sous, John, et al.
Pubblicazione: (2026)

Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
di: Erba, Vittorio, et al.
Pubblicazione: (2024)

The Quantization Model of Neural Scaling
di: Michaud, Eric J., et al.
Pubblicazione: (2023)

A Theory of Saddle Escape in Deep Nonlinear Networks
di: Rawal, Divit, et al.
Pubblicazione: (2026)

High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks
di: Martin, Simon, et al.
Pubblicazione: (2026)

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
di: Defilippis, Leonardo, et al.
Pubblicazione: (2025)

Theory of Speciation Transitions in Diffusion Models with General Class Structure
di: Achilli, Beatrice, et al.
Pubblicazione: (2026)

Dynamical Mean-Field Theory of Self-Attention Neural Networks
di: Poc-López, Ángel, et al.
Pubblicazione: (2024)