Cover Image

On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ishikawa, Satoki, Karakida, Ryo
Format:	Preprint
Published:	2023
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
by: Ishikawa, Satoki, et al.
Published: (2024)

Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
by: Sakai, Mana, et al.
Published: (2025)

Understanding MLP-Mixer as a Wide and Sparse MLP
by: Hayase, Tomohiro, et al.
Published: (2023)

Optimal Layer Selection for Latent Data Augmentation
by: Takase, Tomoumi, et al.
Published: (2024)

Self-attention Networks Localize When QK-eigenspectrum Concentrates
by: Bao, Han, et al.
Published: (2024)

Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models
by: Takeda, Ken, et al.
Published: (2026)

Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
by: Tomihari, Akiyoshi, et al.
Published: (2025)

Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
by: Hayase, Tomohiro, et al.
Published: (2025)

The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks
by: Watanabe, Taishi, et al.
Published: (2026)

A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
by: Hayase, Tomohiro, et al.
Published: (2026)

PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis
by: Ishikawa, Satoki, et al.
Published: (2024)

Channel Estimation by Infinite Width Convolutional Networks
by: Mallik, Mohammed, et al.
Published: (2025)

Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking
by: Karakida, Ryo, et al.
Published: (2024)

Flexible Infinite-Width Graph Convolutional Neural Networks
by: Anson, Ben, et al.
Published: (2024)

Infinite Width Limits of Self Supervised Neural Networks
by: Fleissner, Maximilian, et al.
Published: (2024)

Depth Separation in Norm-Bounded Infinite-Width Neural Networks
by: Parkinson, Suzanna, et al.
Published: (2024)

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network
by: Kim, Taeyoung, et al.
Published: (2023)

Transfer Learning in Infinite Width Feature Learning Networks
by: Lauditi, Clarissa, et al.
Published: (2025)

FOSI: Hybrid First and Second Order Optimization
by: Sivan, Hadar, et al.
Published: (2023)

Second Order Methods for Bandit Optimization and Control
by: Suggala, Arun, et al.
Published: (2024)

Network Degeneracy as an Indicator of Training Performance: Comparing Finite and Infinite Width Angle Predictions
by: Jakub, Cameron, et al.
Published: (2023)

Second-Order Forward-Mode Automatic Differentiation for Optimization
by: Cobb, Adam D., et al.
Published: (2024)

How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences
by: Seleznova, Mariia
Published: (2026)

Comparing BFGS and OGR for Second-Order Optimization
by: Przybysz, Adrian, et al.
Published: (2025)

The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis
by: Pham, Hoang, et al.
Published: (2025)

Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
by: Gomes, Damien Martins
Published: (2025)

On the Infinite Width and Depth Limits of Predictive Coding Networks
by: Innocenti, Francesco, et al.
Published: (2026)

Les Houches Lectures on Deep Learning at Large & Infinite Width
by: Bahri, Yasaman, et al.
Published: (2023)

A Split-Client Approach to Second-Order Optimization
by: Chayti, El Mahdi, et al.
Published: (2025)

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks
by: Yang, Yahong, et al.
Published: (2023)

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
by: Sernau, Luke
Published: (2024)

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
by: Zhao, Yanjun, et al.
Published: (2024)

Fast Deep Hedging with Second-Order Optimization
by: Mueller, Konrad, et al.
Published: (2024)

Lion Cub: Minimizing Communication Overhead in Distributed Lion
by: Ishikawa, Satoki, et al.
Published: (2024)

Scaling Exponents Across Parameterizations and Optimizers
by: Everett, Katie, et al.
Published: (2024)

Finding Differentially Private Second Order Stationary Points in Stochastic Minimax Optimization
by: Xu, Difei, et al.
Published: (2026)

Second-Order Convergence in Private Stochastic Non-Convex Optimization
by: Tao, Youming, et al.
Published: (2025)

SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning
by: Jia, Jinghan, et al.
Published: (2024)

Loss Gradient Gaussian Width based Generalization and Optimization Guarantees
by: Banerjee, Arindam, et al.
Published: (2024)

FUSE: First-Order and Second-Order Unified SynthEsis in Stochastic Optimization
by: Jiang, Zhanhong, et al.
Published: (2025)