Cover Image

Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lu, Yishun, Armour, Wesley
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
by: Garg, Ishir, et al.
Published: (2026)

ONG: Orthogonal Natural Gradient Descent
by: Yadav, Yajat, et al.
Published: (2025)

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
by: Wang, Shaoqi, et al.
Published: (2024)

Optimizing Predictive AI in Physical Design Flows with Mini Pixel Batch Gradient Descent
by: Yang, Haoyu, et al.
Published: (2024)

Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
by: Lu, Yishun, et al.
Published: (2026)

Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
by: Naganuma, Hiroki, et al.
Published: (2026)

Geometrically Inspired Kernel Machines for Collaborative Learning Beyond Gradient Descent
by: Kumar, Mohit, et al.
Published: (2024)

Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation
by: Zhang, Xiyang, et al.
Published: (2026)

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent
by: Liu, Weihua, et al.
Published: (2024)

Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning
by: Huo, Yingxiao, et al.
Published: (2026)

On the Spectral Structure and Objective Equivalence of Orthogonal Multilabel Fisher Discriminants
by: Keith-Norambuena, Brian, et al.
Published: (2026)

Gradient Descent Algorithm Survey
by: Fucheng, Deng, et al.
Published: (2025)

Revisiting LARS for Large Batch Training Generalization of Neural Networks
by: Do, Khoi, et al.
Published: (2023)

Natural Gradient Descent for Online Continual Learning
by: Khawand, Joe, et al.
Published: (2026)

Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training
by: Tyagi, Sahil, et al.
Published: (2026)

Randomness and Interpolation Improve Gradient Descent
by: Li, Jiawen, et al.
Published: (2025)

Learning Associative Memories with Gradient Descent
by: Cabannes, Vivien, et al.
Published: (2024)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
by: Su, DiJia, et al.
Published: (2025)

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training
by: Ma, Yuhan, et al.
Published: (2024)

Orthogonalized Policy Optimization:Policy Optimization as Orthogonal Projection in Hilbert Space
by: Zixian, Wang
Published: (2026)

Adaptive Heavy-Tailed Stochastic Gradient Descent
by: Gong, Bodu, et al.
Published: (2025)

Vanilla Gradient Descent for Oblique Decision Trees
by: Panda, Subrat Prasad, et al.
Published: (2024)

Stochastic Gradient Descent with Momentum is Algorithmically Stable
by: Lei, Yunwen, et al.
Published: (2026)

Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training
by: Lu, Yishun, et al.
Published: (2026)

Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection
by: Yang, Ziyu, et al.
Published: (2026)

Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
by: Zixian, Wang
Published: (2026)

Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
by: Huang, Jia-Hong, et al.
Published: (2024)

Generalized Euler Logarithm and its Applications in Machine Learning: Natural Gradient, Backpropagation, Generalized EG, Mirror Descent and OLPS
by: Cichocki, Andrzej
Published: (2025)

The Initialization Determines Whether In-Context Learning Is Gradient Descent
by: Xie, Shifeng, et al.
Published: (2025)

Elastic Multi-Gradient Descent for Parallel Continual Learning
by: Lyu, Fan, et al.
Published: (2024)

Revisiting the Initial Steps in Adaptive Gradient Descent Optimization
by: Abuduweili, Abulikemu, et al.
Published: (2024)

Efficient Search for Customized Activation Functions with Gradient Descent
by: Strack, Lukas, et al.
Published: (2024)

Conflict-Averse Gradient Descent for Multi-task Learning
by: Liu, Bo, et al.
Published: (2021)

Noise Balance and Stationary Distribution of Stochastic Gradient Descent
by: Ziyin, Liu, et al.
Published: (2023)

Can LLMs predict the convergence of Stochastic Gradient Descent?
by: Zekri, Oussama, et al.
Published: (2024)

Optimizing ML Training with Metagradient Descent
by: Engstrom, Logan, et al.
Published: (2025)

Gradient Descent Efficiency Index
by: Dhingra, Aviral
Published: (2024)

Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
by: Zhao, Shen-Yi, et al.
Published: (2020)

Gradient Regularized Natural Gradients
by: Dash, Satya Prakash, et al.
Published: (2026)

Weighted Stochastic Differential Equation to Implement Wasserstein-Fisher-Rao Gradient Flow
by: Rahimi, Herlock
Published: (2025)