Copertina

LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Robert, Thomas, Safaryan, Mher, Modoranu, Ionut-Vlad, Alistarh, Dan
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
di: Modoranu, Ionut-Vlad, et al.
Pubblicazione: (2026)

FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models
di: Modoranu, Ionut-Vlad, et al.
Pubblicazione: (2025)

MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
di: Modoranu, Ionut-Vlad, et al.
Pubblicazione: (2024)

Error Feedback Can Accurately Compress Preconditioners
di: Modoranu, Ionut-Vlad, et al.
Pubblicazione: (2023)

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
di: Wu, Diyuan, et al.
Pubblicazione: (2024)

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
di: Modoranu, Ionut-Vlad, et al.
Pubblicazione: (2026)

Unified Scaling Laws for Compressed Representations
di: Panferov, Andrei, et al.
Pubblicazione: (2025)

GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
di: Maranjyan, Artavazd, et al.
Pubblicazione: (2022)

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
di: Jovanović, Andrej, et al.
Pubblicazione: (2026)

On Biased Compression for Distributed Learning
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2020)

Towards Robust Scaling Laws for Optimizers
di: Volkova, Alexandra, et al.
Pubblicazione: (2026)

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
di: Tabesh, Soroush, et al.
Pubblicazione: (2025)

ASGO: Adaptive Structured Gradient Optimization
di: An, Kang, et al.
Pubblicazione: (2025)

Random Gradient-Free Optimization in Infinite Dimensional Spaces
di: Peixoto, Caio Lins, et al.
Pubblicazione: (2025)

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
di: Zhou, Dongruo, et al.
Pubblicazione: (2018)

Adaptive Optimization via Momentum on Variance-Normalized Gradients
di: Patitucci, Francisco, et al.
Pubblicazione: (2026)

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems
di: Garber, Dan, et al.
Pubblicazione: (2022)

Low-Rank Mirror-Prox for Nonsmooth and Low-Rank Matrix Optimization Problems
di: Garber, Dan, et al.
Pubblicazione: (2022)

Gradient-Variation Online Adaptivity for Accelerated Optimization with Hölder Smoothness
di: Zhao, Yuheng, et al.
Pubblicazione: (2025)

Low-Rank Extragradient Methods for Scalable Semidefinite Optimization
di: Garber, Dan, et al.
Pubblicazione: (2024)

Directional Smoothness and Gradient Methods: Convergence and Adaptivity
di: Mishkin, Aaron, et al.
Pubblicazione: (2024)

Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
di: Yang, Yufeng, et al.
Pubblicazione: (2024)

Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization
di: Borodich, Ekaterina, et al.
Pubblicazione: (2025)

A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
di: Chakrabarti, Kushal, et al.
Pubblicazione: (2022)

Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
di: Suh, Min-Kook, et al.
Pubblicazione: (2024)

Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learning
di: Li, Yongqi, et al.
Pubblicazione: (2025)

Adaptive Conditional Gradient Descent
di: Khademi, Abbas, et al.
Pubblicazione: (2025)

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods
di: O'Leary-Roseberry, Thomas, et al.
Pubblicazione: (2024)

Adaptive Proximal Gradient Method for Convex Optimization
di: Malitsky, Yura, et al.
Pubblicazione: (2023)

Stochastic Gradient Descent with Adaptive Data
di: Che, Ethan, et al.
Pubblicazione: (2024)

LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM
di: Refael, Yehonathan, et al.
Pubblicazione: (2025)

Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness
di: Crawshaw, Michael, et al.
Pubblicazione: (2025)

Layer-wise Quantization for Quantized Optimistic Dual Averaging
di: Nguyen, Anh Duc, et al.
Pubblicazione: (2025)

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
di: Xie, Shengping, et al.
Pubblicazione: (2025)

Stochastic Adaptive Gradient Descent Without Descent
di: Aujol, Jean-François, et al.
Pubblicazione: (2025)

An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method
di: Zhang, Jiahao, et al.
Pubblicazione: (2024)

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2024)

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
di: Köhne, Frederik, et al.
Pubblicazione: (2023)

Corridor Geometry in Gradient-Based Optimization
di: Dherin, Benoit, et al.
Pubblicazione: (2024)

Differentially Private Optimization with Sparse Gradients
di: Ghazi, Badih, et al.
Pubblicazione: (2024)