Cover Image

On the origin of neural scaling laws: from random graphs to natural language

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Barkeshli, Maissam, Alfarano, Alberto, Gromov, Andrey
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Disordered Systems and Neural Networks Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
by: Kalra, Dayal Singh, et al.
Published: (2026)

Why Warmup the Learning Rate? Underlying Mechanisms and Improvements
by: Kalra, Dayal Singh, et al.
Published: (2024)

Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
by: Tao, Tao, et al.
Published: (2025)

When Can You Get Away with Low Memory Adam?
by: Kalra, Dayal Singh, et al.
Published: (2025)

Towards Distributed Neural Architectures
by: Cowsik, Aditya, et al.
Published: (2025)

Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos
by: Kalra, Dayal Singh, et al.
Published: (2023)

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs
by: Kalra, Dayal Singh, et al.
Published: (2026)

(How) Can Transformers Predict Pseudo-Random Numbers?
by: Tao, Tao, et al.
Published: (2025)

Dimension-free deterministic equivalents and scaling laws for random feature regression
by: Defilippis, Leonardo, et al.
Published: (2024)

Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks
by: D'Amico, Francesco, et al.
Published: (2025)

Identifying internal patterns in (1+1)-dimensional directed percolation using neural networks
by: Parkhomenko, Danil, et al.
Published: (2025)

To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
by: Doshi, Darshil, et al.
Published: (2023)

The Persian Rug: solving toy models of superposition using large-scale symmetries
by: Cowsik, Aditya, et al.
Published: (2024)

A universal compression theory for lottery ticket hypothesis and neural scaling laws
by: Wang, Hong-Yi, et al.
Published: (2025)

An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem
by: Nam, Yoonsoo, et al.
Published: (2024)

Solving excited states for long-range interacting trapped ions with neural networks
by: Ma, Yixuan, et al.
Published: (2025)

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
by: Lauditi, Clarissa, et al.
Published: (2026)

Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model
by: Levi, Noam
Published: (2026)

More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
by: Meir, Sagi, et al.
Published: (2026)

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
by: Defilippis, Leonardo, et al.
Published: (2025)

Generalization through variance: how noise shapes inductive biases in diffusion models
by: Vastola, John J.
Published: (2025)

Grokking vs. Learning: Same Features, Different Encodings
by: Manning-Coe, Dmitry, et al.
Published: (2025)

A Spin Glass Characterization of Neural Networks
by: Li, Jun
Published: (2025)

A method for quantifying the generalization capabilities of generative models for solving Ising models
by: Ma, Qunlong, et al.
Published: (2024)

Applications of Statistical Field Theory in Deep Learning
by: Ringel, Zohar, et al.
Published: (2025)

KAN: Kolmogorov-Arnold Networks
by: Liu, Ziming, et al.
Published: (2024)

How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillator
by: Kantamneni, Subhash, et al.
Published: (2024)

A Geometric Perspective on the Difficulties of Learning GNN-based SAT Solvers
by: Skenderi, Geri
Published: (2025)

Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning
by: Staats, Max, et al.
Published: (2023)

Representation Learning on a Random Lattice
by: Brill, Aryeh
Published: (2025)

Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics
by: Avidan, Yehonatan, et al.
Published: (2023)

Parameter Symmetry Potentially Unifies Deep Learning Theory
by: Ziyin, Liu, et al.
Published: (2025)

Smooth Kolmogorov Arnold networks enabling structural knowledge representation
by: Samadi, Moein E., et al.
Published: (2024)

Dynamics of neural scaling laws in random feature regression with powerlaw-distributed kernel eigenvalues
by: Kramp, Jakob, et al.
Published: (2026)

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
by: Marino, Raffaele, et al.
Published: (2023)

Statistical physics analysis of graph neural networks: Approaching optimality in the contextual stochastic block model
by: Duranthon, O., et al.
Published: (2025)

Deterministic versus stochastic dynamical classifiers: opposing random adversarial attacks with noise
by: Chicchi, Lorenzo, et al.
Published: (2024)

A simple theory for training response of deep neural networks
by: Nakazato, Kenichi
Published: (2024)

Predictive Coding Graphs are a Superset of Feedforward Neural Networks
by: van Zwol, Björn
Published: (2026)

Approximation Theory for Neural Networks: Old and New
by: Mukherjee, Soumendu Sundar, et al.
Published: (2026)