Imagen de Portada

From $O(mn)$ to $O(r^2)$: Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Dang, Sizhe, Shao, Jiaqi, Zheng, Xiaodong, Dai, Guang, Song, Yan, Ye, Haishan
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
por: Dang, Sizhe, et al.
Publicado: (2025)

ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
por: Sun, Zhishen, et al.
Publicado: (2026)

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
por: Zhao, Yanjun, et al.
Publicado: (2024)

Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithms on Smooth Functions
por: Ye, Haishan
Publicado: (2025)

MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
por: Sun, Zhishen, et al.
Publicado: (2025)

Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithm on Stochastic Smooth Functions
por: Ye, Haishan
Publicado: (2025)

Optimal High-Probability Regret for Online Convex Optimization with Two-Point Bandit Feedback
por: Ye, Haishan
Publicado: (2026)

Breaking the O(mn)-Time Barrier for Vertex-Weighted Global Minimum Cut
por: Chuzhoy, Julia, et al.
Publicado: (2025)

Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injection
por: Chang, Xiangyu, et al.
Publicado: (2025)

On the Convergence of Single-Loop Stochastic Bilevel Optimization with Approximate Implicit Differentiation
por: Zhou, Yubo, et al.
Publicado: (2026)

Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
por: Sun, Zhishen, et al.
Publicado: (2025)

Additive One Approximation for Minimum Degree Spanning Tree: Breaking the $O(mn)$ Time Barrier
por: Bhattacharya, Sayan, et al.
Publicado: (2026)

High-Probability Guarantees for Random Zeroth-Order Gradient Descent on Smooth Functions
por: Ye, Haishan
Publicado: (2026)

High-Probability Guarantees for Random Zeroth-Order (Stochastic) Gradient Descent
por: Ye, Haishan
Publicado: (2026)

AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning
por: Coquelin, Daniel, et al.
Publicado: (2024)

Convergence Rate Analysis of the AdamW-Style Shampoo: Unifying One-Sided and Two-Sided Preconditioning
por: Li, Huan, et al.
Publicado: (2026)

Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices
por: Zhao, Pengxiang, et al.
Publicado: (2024)

Near-Optimal Distributed Minimax Optimization under the Second-Order Similarity
por: Zhou, Qihao, et al.
Publicado: (2024)

Photocatalytic Radical‐Polar Crossover Enables Modular Access to Bicyclo[2.m.n]Alkane Alcohol Bioisosteres
por: Hui Ran, et al.
Publicado: (2026)

Riemannian Momentum Tracking: Distributed Optimization with Momentum on Compact Submanifolds
por: Chen, Jun, et al.
Publicado: (2026)

Low-Communication Resilient Distributed Estimation Algorithm Based on Memory Mechanism
por: Li, Wei, et al.
Publicado: (2025)

FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models
por: Shao, Zishan, et al.
Publicado: (2025)

From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
por: Chen, Zheng-An, et al.
Publicado: (2025)

Zero-Order Sharpness-Aware Minimization
por: Fu, Yao, et al.
Publicado: (2025)

cMPI: Using CXL Memory Sharing for MPI One-Sided and Two-Sided Inter-Node Communications
por: Wang, Xi, et al.
Publicado: (2025)

Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning
por: Waykole, Vedant, et al.
Publicado: (2026)

Fairness and Efficiency in Two-Sided Matching Markets
por: Jain, Pallavi, et al.
Publicado: (2025)

When Can You Get Away with Low Memory Adam?
por: Kalra, Dayal Singh, et al.
Publicado: (2025)

Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient
por: Di, Hao, et al.
Publicado: (2024)

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
por: Wei, Xiuying, et al.
Publicado: (2024)

Optimal Decentralized Composite Optimization for Convex Functions
por: Ye, Haishan, et al.
Publicado: (2023)

Why Does Adaptive Zeroth-Order Optimization Work?
por: Ye, Haishan, et al.
Publicado: (2026)

Stochastic Diagonal Estimation Based on Matrix Quadratic Form Oracles
por: Ye, Haishan, et al.
Publicado: (2025)

Can a One-Point Feedback Zeroth-order Algorithm Achieve Linear Dimension Dependent Sample Complexity?
por: Ye, Haishan, et al.
Publicado: (2025)

Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
por: Wang, Yezhen, et al.
Publicado: (2025)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
por: Zhao, Jiawei, et al.
Publicado: (2024)

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
por: Jovanović, Andrej, et al.
Publicado: (2026)

High‐Performance Monolayer 1T‐GeO 2 Transistors with Low‐Resistance Metal Contacts
por: Shuai Lang, et al.
Publicado: (2025)

Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
por: Zheng, Estelle, et al.
Publicado: (2025)

AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reuse
por: Yu, Zichao, et al.
Publicado: (2025)