Copertina

Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Sun, Ruotong, Wei, Ermin
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
di: Eschenhagen, Runa, et al.
Pubblicazione: (2025)

In-Context Reward Adaptation for Robust Preference Modeling
di: Sun, Zhenyu, et al.
Pubblicazione: (2026)

SOAP: Improving and Stabilizing Shampoo using Adam
di: Vyas, Nikhil, et al.
Pubblicazione: (2024)

4-bit Shampoo for Memory-Efficient Network Training
di: Wang, Sike, et al.
Pubblicazione: (2024)

Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
di: Eschenhagen, Runa, et al.
Pubblicazione: (2026)

Data Whitening Improves Sparse Autoencoder Learning
di: Saraswatula, Ashwin, et al.
Pubblicazione: (2025)

Orthogonalized Policy Optimization:Policy Optimization as Orthogonal Projection in Hilbert Space
di: Zixian, Wang
Pubblicazione: (2026)

Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
di: Zixian, Wang
Pubblicazione: (2026)

SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
di: Ma, Chao, et al.
Pubblicazione: (2024)

Whitening Not Recommended for Classification Tasks in LLMs
di: Forooghi, Ali, et al.
Pubblicazione: (2024)

Hessian-Free Online Certified Unlearning
di: Qiao, Xinbao, et al.
Pubblicazione: (2024)

Reservoir Subspace Injection for Online ICA under Top-n Whitening
di: Xiao, Wenjun, et al.
Pubblicazione: (2026)

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
di: Nam, Kyunghun, et al.
Pubblicazione: (2026)

Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
di: Garg, Ishir, et al.
Pubblicazione: (2026)

Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
di: Yao, Wei, et al.
Pubblicazione: (2025)

Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
di: Shu, Yao, et al.
Pubblicazione: (2026)

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
di: Rahulamathavan, Yogachandran, et al.
Pubblicazione: (2026)

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
di: Wang, Shaoqi, et al.
Pubblicazione: (2024)

Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training
di: Lu, Yishun, et al.
Pubblicazione: (2025)

GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
di: Xiang, Maoyang, et al.
Pubblicazione: (2026)

Deep Orthogonal Hypersphere Compression for Anomaly Detection
di: Zhang, Yunhe, et al.
Pubblicazione: (2023)

Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics
di: Zhang, Wenqing, et al.
Pubblicazione: (2024)

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
di: Lin, Bokai, et al.
Pubblicazione: (2024)

Multiplicative Orthogonal Sequential Editing for Language Models
di: Xu, Hao-Xiang, et al.
Pubblicazione: (2026)

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
di: He, Junlin, et al.
Pubblicazione: (2024)

Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
di: Yang, Hanlin, et al.
Pubblicazione: (2024)

ROOT: Robust Orthogonalized Optimizer for Neural Network Training
di: He, Wei, et al.
Pubblicazione: (2025)

Recovering implicit physics model under real-world constraints
di: Banerjee, Ayan, et al.
Pubblicazione: (2024)

Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
di: Zhang, Yixian, et al.
Pubblicazione: (2025)

Scalable Heterogeneous Graph Learning via Heterogeneous-aware Orthogonal Prototype Experts
di: Zhou, Wei, et al.
Pubblicazione: (2026)

Pre-training with Random Orthogonal Projection Image Modeling
di: Haghighat, Maryam, et al.
Pubblicazione: (2023)

Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection
di: Yang, Ziyu, et al.
Pubblicazione: (2026)

KL for a KL: On-Policy Distillation with Control Variate Baseline
di: Oh, Minjae, et al.
Pubblicazione: (2026)

Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?
di: Blum, Avrim, et al.
Pubblicazione: (2019)

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
di: Schlaginhaufen, Andreas, et al.
Pubblicazione: (2024)

Optimal Stability of KL Divergence under Gaussian Perturbations
di: Pan, Jialu, et al.
Pubblicazione: (2026)

KL-regularization Itself is Differentially Private in Bandits and RLHF
di: Zhang, Yizhou, et al.
Pubblicazione: (2025)

ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
di: Schröder, Maresa, et al.
Pubblicazione: (2026)

Measuring Orthogonality in Representations of Generative Models
di: Geyer, Robin C., et al.
Pubblicazione: (2024)

ONG: Orthogonal Natural Gradient Descent
di: Yadav, Yajat, et al.
Pubblicazione: (2025)