Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters

Documenti analoghi

• Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
  di: Choudhury, Sayantan, et al.
  Pubblicazione: (2026)
• Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity
  di: Gorbunov, Eduard, et al.
  Pubblicazione: (2024)
• Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
  di: Chezhegov, Savelii, et al.
  Pubblicazione: (2024)
• AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
  di: Lau, Tim Tsz-Kit, et al.
  Pubblicazione: (2024)
• Parameter-free Clipped Gradient Descent Meets Polyak
  di: Takezawa, Yuki, et al.
  Pubblicazione: (2024)