Cover Image

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Fei, Shen, Li, Ding, Liang, Xue, Chao, Liu, Ye, Ding, Changxing
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
by: Wang, Fei, et al.
Published: (2026)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
by: Zhang, Yihua, et al.
Published: (2024)

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
by: Ling, Zhenqing, et al.
Published: (2024)

Prior-Informed Zeroth-Order Optimization with Adaptive Direction Alignment for Memory-Efficient LLM Fine-Tuning
by: Jin, Feihu, et al.
Published: (2026)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
by: Yang, Yifan, et al.
Published: (2024)

Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
by: Wang, Fei, et al.
Published: (2025)

Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
by: Li, Zeman, et al.
Published: (2024)

On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
by: Katti, Prabodh, et al.
Published: (2025)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
by: Li, Zelin, et al.
Published: (2026)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
by: Gautam, Tanmay, et al.
Published: (2024)

Private Fine-tuning of Large Language Models with Zeroth-order Optimization
by: Tang, Xinyu, et al.
Published: (2024)

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
by: Sun, Yan, et al.
Published: (2025)

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
by: Ghiasvand, Sajjad, et al.
Published: (2024)

Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning
by: Liu, Z, et al.
Published: (2024)

Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
by: Jie, Shibo, et al.
Published: (2024)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
by: Liu, Yong, et al.
Published: (2024)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
by: Zhou, Jiajun, et al.
Published: (2025)

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
by: Dang, Sizhe, et al.
Published: (2025)

Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
by: Zhang, Kairun, et al.
Published: (2025)

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
by: Liu, Yongkang, et al.
Published: (2026)

ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
by: Wang, Liangyu, et al.
Published: (2025)

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
by: Tan, Qitao, et al.
Published: (2025)

Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
by: Jin, Feihu, et al.
Published: (2026)

Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection
by: Shen, Lingzhi, et al.
Published: (2025)

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
by: Pan, Rui, et al.
Published: (2024)

MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models
by: Zhang, Zhen, et al.
Published: (2025)

TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Models
by: Liu, Zefang, et al.
Published: (2024)

SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
by: Lu, Xudong, et al.
Published: (2024)

Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models
by: Liao, Xiaoxuan, et al.
Published: (2025)

Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networks
by: Li, Zuguang, et al.
Published: (2024)

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
by: Zhao, Yanjun, et al.
Published: (2024)

Memory-Efficient Fine-Tuning of Transformers via Token Selection
by: Simoulin, Antoine, et al.
Published: (2025)

Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution
by: Tan, Wentao, et al.
Published: (2024)

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
by: Seung, Hyunseok, et al.
Published: (2025)

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models
by: Wu, Yichao, et al.
Published: (2024)

Crafting Efficient Fine-Tuning Strategies for Large Language Models
by: Oliver, Michael, et al.
Published: (2024)

Fine-Tuning Language Models with Reward Learning on Policy
by: Lang, Hao, et al.
Published: (2024)

Efficient End-to-end Language Model Fine-tuning on Graphs
by: Xue, Rui, et al.
Published: (2023)

A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models
by: Gu, Jian, et al.
Published: (2024)

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models
by: Doering, Nigel, et al.
Published: (2024)