Imagen de Portada

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Cai, Zhijie, Chen, Haolong, Zhu, Guangxu
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
por: Liu, Yong, et al.
Publicado: (2024)

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
por: Wang, Shuo, et al.
Publicado: (2026)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
por: Zhou, Jiajun, et al.
Publicado: (2025)

MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models
por: Zhang, Zhen, et al.
Publicado: (2025)

AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
por: Chen, Ziye, et al.
Publicado: (2026)

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
por: Dang, Sizhe, et al.
Publicado: (2025)

STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction
por: Chen, Haolong, et al.
Publicado: (2025)

OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning
por: Long, Jikai, et al.
Publicado: (2025)

Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizer
por: Feng, Jian, et al.
Publicado: (2026)

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
por: Sun, Yan, et al.
Publicado: (2025)

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
por: Zhao, Huaqin, et al.
Publicado: (2024)

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
por: Guo, Wentao, et al.
Publicado: (2024)

Three Birds, One Stone: Solving the Communication-Memory-Privacy Trilemma in LLM Fine-tuning Over Wireless Networks with Zeroth-Order Optimization
por: Cai, Zhijie, et al.
Publicado: (2026)

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
por: Ran, Yide, et al.
Publicado: (2025)

Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
por: Yu, Ziming, et al.
Publicado: (2024)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
por: Yang, Yifan, et al.
Publicado: (2024)

Refining Adaptive Zeroth-Order Optimization at Ease
por: Shu, Yao, et al.
Publicado: (2025)

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
por: Tan, Qitao, et al.
Publicado: (2025)

ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
por: Meng, Chuiyang, et al.
Publicado: (2026)

Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning
por: Chen, Yutong, et al.
Publicado: (2025)

Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
por: Kim, Minseon, et al.
Publicado: (2025)

Subspace-based Approximate Hessian Method for Zeroth-Order Optimization
por: Kim, Dongyoon, et al.
Publicado: (2025)

Zeroth-Order Optimization Finds Flat Minima
por: Zhang, Liang, et al.
Publicado: (2025)

DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
por: Wang, Liangyu, et al.
Publicado: (2025)

Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
por: Luo, Yu, et al.
Publicado: (2026)

A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learning
por: Wu, Chenlin, et al.
Publicado: (2024)

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles
por: Tang, Zhiwei, et al.
Publicado: (2023)

On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization
por: Ma, Shaocong, et al.
Publicado: (2025)

Order-Independence Without Fine Tuning
por: McIlroy-Young, Reid, et al.
Publicado: (2024)

ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm
por: Xie, Lunchen, et al.
Publicado: (2025)

FeedSign: Robust Full-parameter Federated Fine-tuning of Large Models with Extremely Low Communication Overhead of One Bit
por: Cai, Zhijie, et al.
Publicado: (2025)

Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization
por: Liang, Chen, et al.
Publicado: (2026)

AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization
por: Cemri, Mert, et al.
Publicado: (2026)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
por: Gautam, Tanmay, et al.
Publicado: (2024)

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
por: Fan, Jiajun, et al.
Publicado: (2025)

S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
por: Yang, Xinyu, et al.
Publicado: (2024)

ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization
por: Sugiura, Keisuke, et al.
Publicado: (2025)

Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment
por: Wang, Haozhong, et al.
Publicado: (2026)

The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective
por: Liang, Qiyao, et al.
Publicado: (2026)

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
por: Legate, Gwen, et al.
Publicado: (2025)