Copertina

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhang, Yihua, Li, Pingzhi, Hong, Junyuan, Li, Jiaxiang, Zhang, Yimeng, Zheng, Wenqing, Chen, Pin-Yu, Lee, Jason D., Yin, Wotao, Hong, Mingyi, Wang, Zhangyang, Liu, Sijia, Chen, Tianlong
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
di: Lang, Yicheng, et al.
Pubblicazione: (2026)

DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training
di: Chen, Aochuan, et al.
Pubblicazione: (2023)

One Token Embedding Is Enough to Deadlock Your Large Reasoning Model
di: Zhang, Mohan, et al.
Pubblicazione: (2025)

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
di: Guo, Wentao, et al.
Pubblicazione: (2024)

A Note on the Convergence of Muon
di: Li, Jiaxiang, et al.
Pubblicazione: (2025)

Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Risk
di: Li, Zhangheng, et al.
Pubblicazione: (2024)

Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
di: Zhang, Kairun, et al.
Pubblicazione: (2025)

MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
di: Pandit, Shrey, et al.
Pubblicazione: (2025)

Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
di: Glentis, Athanasios, et al.
Pubblicazione: (2025)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
di: Li, Zelin, et al.
Pubblicazione: (2026)

A Doubly Stochastically Perturbed Algorithm for Linearly Constrained Bilevel Optimization
di: Khanduri, Prashant, et al.
Pubblicazione: (2025)

Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models
di: Zhang, Yimeng, et al.
Pubblicazione: (2024)

LAD-VF: LLM-Automatic Differentiation Enables Fine-Tuning-Free Robot Planning from Formal Methods Feedback
di: Yang, Yunhao, et al.
Pubblicazione: (2025)

Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
di: Yu, Ziming, et al.
Pubblicazione: (2024)

Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
di: Li, Zeman, et al.
Pubblicazione: (2024)

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
di: Wang, Fei, et al.
Pubblicazione: (2024)

Problem-Parameter-Free Decentralized Nonconvex Stochastic Optimization
di: Li, Jiaxiang, et al.
Pubblicazione: (2024)

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
di: Liu, Sijia, et al.
Pubblicazione: (2026)

A Barrier Function Approach for Bilevel Optimization with Coupled Lower-Level Constraints: Formulation, Approximation and Algorithms
di: Jiang, Xiaotian, et al.
Pubblicazione: (2024)

The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection
di: Jafari, Mohammad, et al.
Pubblicazione: (2024)

Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
di: Glentis, Athanasios, et al.
Pubblicazione: (2025)

Enhancing Quantum Security over Federated Learning via Post-Quantum Cryptography
di: Li, Pingzhi, et al.
Pubblicazione: (2024)

Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond
di: Fan, Chongyu, et al.
Pubblicazione: (2025)

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
di: Perin, Gabriel J., et al.
Pubblicazione: (2025)

Make Optimization Once and for All with Fine-grained Guidance
di: Shi, Mingjia, et al.
Pubblicazione: (2025)

Towards Building Non-Fine-Tunable Foundation Models
di: Wang, Ziyao, et al.
Pubblicazione: (2026)

ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
di: Peng, Yuezhang, et al.
Pubblicazione: (2025)

SEAL: Steerable Reasoning Calibration of Large Language Models for Free
di: Chen, Runjin, et al.
Pubblicazione: (2025)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
di: Yang, Yifan, et al.
Pubblicazione: (2024)

First-Order Algorithms Without Lipschitz Gradient: A Sequential Local Optimization Approach
di: Zhang, Junyu, et al.
Pubblicazione: (2020)

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
di: Ling, Zhenqing, et al.
Pubblicazione: (2024)

MindOpt Adapter for CPLEX Benchmarking Performance Analysis
di: Sun, Mou, et al.
Pubblicazione: (2023)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
di: Zhou, Jiajun, et al.
Pubblicazione: (2025)

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
di: Ran, Yide, et al.
Pubblicazione: (2025)

Prior-Informed Zeroth-Order Optimization with Adaptive Direction Alignment for Memory-Efficient LLM Fine-Tuning
di: Jin, Feihu, et al.
Pubblicazione: (2026)

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning
di: Chen, Yiwei, et al.
Pubblicazione: (2025)

More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
di: Wang, Yifan, et al.
Pubblicazione: (2025)

Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
di: Qiu, Junbin, et al.
Pubblicazione: (2026)

A Correspondence-Driven Approach for Bilevel Decision-making with Nonconvex Lower-Level Problems
di: Jiang, Xiaotian, et al.
Pubblicazione: (2025)

How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance
di: Li, Hongkang, et al.
Pubblicazione: (2024)