Copertina

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Garrod, Connall, Keating, Jonathan P., Thrampoulidis, Christos
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

The Persistence of Neural Collapse Despite Low-Rank Bias
di: Garrod, Connall, et al.
Pubblicazione: (2024)

Unifying Low Dimensional Spectra in Deep Learning
di: Garrod, Connall, et al.
Pubblicazione: (2024)

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
di: Fan, Chen, et al.
Pubblicazione: (2025)

Implicit Bias and Fast Convergence Rates for Self-attention
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2024)

On the Optimization and Generalization of Multi-head Attention
di: Deora, Puneesh, et al.
Pubblicazione: (2023)

Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions
di: Varre, Aditya, et al.
Pubblicazione: (2026)

Memory capacity of two layer neural networks with smooth activations
di: Madden, Liam, et al.
Pubblicazione: (2023)

Next-token prediction capacity: general upper bounds and a lower bound for transformers
di: Madden, Liam, et al.
Pubblicazione: (2024)

Transformers as Support Vector Machines
di: Tarzanagh, Davoud Ataee, et al.
Pubblicazione: (2023)

Tractable Representations for Convergent Approximation of Distributional HJB Equations
di: Alhosh, Julie, et al.
Pubblicazione: (2025)

Graph Similarity Regularized Softmax for Semi-Supervised Node Classification
di: Yang, Yiming, et al.
Pubblicazione: (2024)

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods
di: Klein, Sara, et al.
Pubblicazione: (2023)

Achieving Tractable Minimax Optimal Regret in Average Reward MDPs
di: Boone, Victor, et al.
Pubblicazione: (2024)

Riemannian Optimization for Hadamard Products of Low-Rank Matrices
di: Jawanpuria, Pratik, et al.
Pubblicazione: (2026)

Efficient Sparse PCA via Block-Diagonalization
di: Del Pia, Alberto, et al.
Pubblicazione: (2024)

Optimization Insights into Deep Diagonal Linear Networks
di: Labarrière, Hippolyte, et al.
Pubblicazione: (2024)

On Dissipativity of Cross-Entropy Loss in Training ResNets
di: Püttschneider, Jens, et al.
Pubblicazione: (2024)

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
di: Kolb, Chris, et al.
Pubblicazione: (2023)

Efficient algorithms for the Hadamard decomposition
di: Wertz, Samuel, et al.
Pubblicazione: (2025)

Online Optimization on Hadamard Manifolds: Curvature Independent Regret Bounds on Horospherically Convex Objectives
di: Sahinoglu, Emre, et al.
Pubblicazione: (2025)

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
di: Papazov, Hristo, et al.
Pubblicazione: (2024)

Implicit Regularization of Gradient Flow on One-Layer Softmax Attention
di: Sheen, Heejune, et al.
Pubblicazione: (2024)

Tractable hierarchies of convex relaxations for polynomial optimization on the nonnegative orthant
di: Mai, Ngoc Hoang Anh, et al.
Pubblicazione: (2022)

Conditional Risk Minimization with Side Information: A Tractable, Universal Optimal Transport Framework
di: Xie, Xinqiao, et al.
Pubblicazione: (2025)

Decentralized Online Riemannian Optimization Beyond Hadamard Manifolds
di: Sahinoglu, Emre, et al.
Pubblicazione: (2025)

Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
di: Chen, Siyu, et al.
Pubblicazione: (2024)

Deconstructing the Goldilocks Zone of Neural Network Initialization
di: Vysogorets, Artem, et al.
Pubblicazione: (2024)

Contextual Distributionally Robust Optimization with Causal and Continuous Structure: An Interpretable and Tractable Approach
di: Zhang, Fenglin, et al.
Pubblicazione: (2026)

High-dimensional Limit of SGD for Diagonal Linear Networks
di: Malaxechebarría, Begoña García, et al.
Pubblicazione: (2026)

Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completion
di: Ma, Jianhao, et al.
Pubblicazione: (2024)

Sample Complexity of Linear Quadratic Regulator Without Initial Stability
di: Moghaddam, Amirreza Neshaei, et al.
Pubblicazione: (2025)

Global Convergence of Four-Layer Matrix Factorization under Random Initialization
di: Luo, Minrui, et al.
Pubblicazione: (2025)

Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations
di: Kumar, Akshay, et al.
Pubblicazione: (2024)

Variational Entropy Search for Adjusting Expected Improvement
di: Cheng, Nuojin, et al.
Pubblicazione: (2024)

Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networks
di: Kumar, Akshay, et al.
Pubblicazione: (2024)

On Adaptivity in Zeroth-Order Optimization
di: Dbouk, Hassan, et al.
Pubblicazione: (2026)

Manifold-based Algorithms for the Hadamard Decomposition
di: Gillis, Nicolas, et al.
Pubblicazione: (2026)

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization
di: Guo, Xin, et al.
Pubblicazione: (2023)

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
di: Das, Mohua, et al.
Pubblicazione: (2026)

A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization
di: Cheng, Nuojin, et al.
Pubblicazione: (2025)