Cover Image

Next-token prediction capacity: general upper bounds and a lower bound for transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Madden, Liam, Fox, Curtis, Thrampoulidis, Christos
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Optimization and Control 15A03, 26B35
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Memory capacity of two layer neural networks with smooth activations
by: Madden, Liam, et al.
Published: (2023)

Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons
by: Madden, Liam
Published: (2024)

Free group of Hamel bijections of big size
by: Natkaniec, Tomasz
Published: (2024)

Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent
by: Cai, HanQin, et al.
Published: (2024)

Escaping Local Minima Provably in Non-convex Matrix Sensing: A Deterministic Framework via Simulated Lifting
by: Shen, Tianqi, et al.
Published: (2026)

Formalizing Pfaffian in bounded arithmetic
by: Kuroda, Satoru
Published: (2024)

The joint spectral radius is pointwise Hölder continuous
by: Epperlein, Jeremias, et al.
Published: (2023)

Upper and lower bounds for the Lipschitz constant of random neural networks
by: Geuchen, Paul, et al.
Published: (2023)

Probabilistic bounds on best rank-one approximation ratio
by: Kozhasov, Khazhgali, et al.
Published: (2022)

Using orthogonally structured positive bases for constructing positive $k$-spanning sets with cosine measure guarantees
by: Hare, Warren, et al.
Published: (2023)

On Bias and Its Reduction via Standardization in Discretized Electromagnetic Source Localization Problems
by: Lahtinen, Joonas
Published: (2024)

Comments and extensions on "State-equivalent form and minimum-order compensator design for rectangular descriptor systems"
by: Shi, Shuo, et al.
Published: (2025)

A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization
by: Chen, Po, et al.
Published: (2025)

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
by: Fan, Chen, et al.
Published: (2025)

Implicit Bias and Fast Convergence Rates for Self-attention
by: Vasudeva, Bhavya, et al.
Published: (2024)

SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
by: Kranz, Julian, et al.
Published: (2025)

Roots, trace, and extendability of flat nonnegative smooth functions
by: Jiang, Fushuai
Published: (2023)

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
by: Garrod, Connall, et al.
Published: (2025)

Optimization without Retraction on the Random Generalized Stiefel Manifold
by: Vary, Simon, et al.
Published: (2024)

A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Properties
by: Qiu, Junwen, et al.
Published: (2023)

A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimization
by: Qiu, Junwen, et al.
Published: (2023)

Lagrange duality on DC evenly convex optimization problems via a generalized conjugation scheme
by: Fajardo, M. D., et al.
Published: (2024)

On the Characterization of gH-partial derivatives and gH-Product for Interval-Valued Functions
by: Suhail, Amir, et al.
Published: (2025)

On the Optimization and Generalization of Multi-head Attention
by: Deora, Puneesh, et al.
Published: (2023)

Matroid adjoints and the minimum rank of zero-nonzero matrix patterns
by: Deaett, Louis, et al.
Published: (2025)

Forcing upper $Σ$-uniformization in the presence of lower $Π$-reduction or uniformization
by: Hoffelner, Stefan
Published: (2025)

Attainable set for rank 3 step 2 free Carnot group with positive controls
by: Podobryaev, A. V.
Published: (2022)

Operator Learning for Families of Finite-State Mean-Field Games
by: Hofgard, William, et al.
Published: (2026)

The distributional divergence of horizontal vector fields vanishing at infinity on Carnot groups
by: Baldi, Annalisa, et al.
Published: (2022)

On the controllability of nonlinear systems with a periodic drift
by: Caillau, Jean-Baptiste, et al.
Published: (2024)

Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs
by: Gupta, Abhishek, et al.
Published: (2026)

Ordinal Analysis of Well-Ordering Principles, Well Quasi-Orders Closure Properties, and $Σ_n$-Collection Schema
by: Buriola, Gabriele, et al.
Published: (2025)

Stationary Mean-Field Games of Singular Control under Knightian Uncertainty
by: Ferrari, Giorgio, et al.
Published: (2025)

Vector Spaces of Linearizations for Multivariable State-Space Systems
by: Bist, Avisek, et al.
Published: (2024)

Liouville Type Theorem for the Steady Fractional Compressible MHD Equations in $\mathbb{R}^{3}$
by: Liu, Zhenyuan, et al.
Published: (2026)

Algebraic Riccati Tensor Equations with Applications in Multilinear Control Systems
by: Wang, Yuchao, et al.
Published: (2024)

A smoothing proximal gradient algorithm for matrix rank minimization problem
by: Yu, Quan, et al.
Published: (2021)

SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon
by: Zhang, Hengrui, et al.
Published: (2026)

Linear regression with overparameterized linear neural networks: Tight upper and lower bounds for implicit $\ell^1$-regularization
by: Matt, Hannes, et al.
Published: (2025)

Strict rearrangement inequalities: nonexpansivity and periodic Gagliardo seminorms
by: Csató, Gyula, et al.
Published: (2024)