Image de couverture de livre

Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sun, Ruotong, Wei, Ermin
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
par: Eschenhagen, Runa, et autres
Publié: (2025)

In-Context Reward Adaptation for Robust Preference Modeling
par: Sun, Zhenyu, et autres
Publié: (2026)

SOAP: Improving and Stabilizing Shampoo using Adam
par: Vyas, Nikhil, et autres
Publié: (2024)

4-bit Shampoo for Memory-Efficient Network Training
par: Wang, Sike, et autres
Publié: (2024)

Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
par: Eschenhagen, Runa, et autres
Publié: (2026)

Data Whitening Improves Sparse Autoencoder Learning
par: Saraswatula, Ashwin, et autres
Publié: (2025)

Orthogonalized Policy Optimization:Policy Optimization as Orthogonal Projection in Hilbert Space
par: Zixian, Wang
Publié: (2026)

Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
par: Zixian, Wang
Publié: (2026)

SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
par: Ma, Chao, et autres
Publié: (2024)

Whitening Not Recommended for Classification Tasks in LLMs
par: Forooghi, Ali, et autres
Publié: (2024)

Hessian-Free Online Certified Unlearning
par: Qiao, Xinbao, et autres
Publié: (2024)

Reservoir Subspace Injection for Online ICA under Top-n Whitening
par: Xiao, Wenjun, et autres
Publié: (2026)

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
par: Nam, Kyunghun, et autres
Publié: (2026)

Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
par: Garg, Ishir, et autres
Publié: (2026)

Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
par: Yao, Wei, et autres
Publié: (2025)

Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
par: Shu, Yao, et autres
Publié: (2026)

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
par: Rahulamathavan, Yogachandran, et autres
Publié: (2026)

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
par: Wang, Shaoqi, et autres
Publié: (2024)

Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training
par: Lu, Yishun, et autres
Publié: (2025)

GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
par: Xiang, Maoyang, et autres
Publié: (2026)

Deep Orthogonal Hypersphere Compression for Anomaly Detection
par: Zhang, Yunhe, et autres
Publié: (2023)

Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics
par: Zhang, Wenqing, et autres
Publié: (2024)

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
par: Lin, Bokai, et autres
Publié: (2024)

Multiplicative Orthogonal Sequential Editing for Language Models
par: Xu, Hao-Xiang, et autres
Publié: (2026)

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
par: He, Junlin, et autres
Publié: (2024)

Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
par: Yang, Hanlin, et autres
Publié: (2024)

ROOT: Robust Orthogonalized Optimizer for Neural Network Training
par: He, Wei, et autres
Publié: (2025)

Recovering implicit physics model under real-world constraints
par: Banerjee, Ayan, et autres
Publié: (2024)

Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
par: Zhang, Yixian, et autres
Publié: (2025)

Scalable Heterogeneous Graph Learning via Heterogeneous-aware Orthogonal Prototype Experts
par: Zhou, Wei, et autres
Publié: (2026)

Pre-training with Random Orthogonal Projection Image Modeling
par: Haghighat, Maryam, et autres
Publié: (2023)

Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection
par: Yang, Ziyu, et autres
Publié: (2026)

KL for a KL: On-Policy Distillation with Control Variate Baseline
par: Oh, Minjae, et autres
Publié: (2026)

Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?
par: Blum, Avrim, et autres
Publié: (2019)

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
par: Schlaginhaufen, Andreas, et autres
Publié: (2024)

Optimal Stability of KL Divergence under Gaussian Perturbations
par: Pan, Jialu, et autres
Publié: (2026)

KL-regularization Itself is Differentially Private in Bandits and RLHF
par: Zhang, Yizhou, et autres
Publié: (2025)

ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
par: Schröder, Maresa, et autres
Publié: (2026)

Measuring Orthogonality in Representations of Generative Models
par: Geyer, Robin C., et autres
Publié: (2024)

ONG: Orthogonal Natural Gradient Descent
par: Yadav, Yajat, et autres
Publié: (2025)