Imagen de Portada

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Tan, Qitao, Liu, Jun, Zhan, Zheng, Ding, Caiwei, Wang, Yanzhi, Ma, Xiaolong, Lee, Jaewoo, Lu, Jin, Yuan, Geng
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Artificial Intelligence Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
por: Mi, Zhendong, et al.
Publicado: (2025)

Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training
por: Tan, Qitao, et al.
Publicado: (2025)

KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning
por: Mi, Zhendong, et al.
Publicado: (2025)

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
por: Seung, Hyunseok, et al.
Publicado: (2025)

Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
por: Chen, Yiming, et al.
Publicado: (2024)

Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
por: Shang, Sifeng, et al.
Publicado: (2025)

Prior-Informed Zeroth-Order Optimization with Adaptive Direction Alignment for Memory-Efficient LLM Fine-Tuning
por: Jin, Feihu, et al.
Publicado: (2026)

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
por: Zhao, Huaqin, et al.
Publicado: (2024)

Private Fine-tuning of Large Language Models with Zeroth-order Optimization
por: Tang, Xinyu, et al.
Publicado: (2024)

Data-efficient LLM Fine-tuning for Code Generation
por: Lv, Weijie, et al.
Publicado: (2025)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
por: Li, Zelin, et al.
Publicado: (2026)

Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deployment
por: Tan, Qitao, et al.
Publicado: (2026)

Fast and Memory-Efficient Video Diffusion Using Streamlined Inference
por: Zhan, Zheng, et al.
Publicado: (2024)

ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
por: Meng, Chuiyang, et al.
Publicado: (2026)

Three Birds, One Stone: Solving the Communication-Memory-Privacy Trilemma in LLM Fine-tuning Over Wireless Networks with Zeroth-Order Optimization
por: Cai, Zhijie, et al.
Publicado: (2026)

Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
por: Jin, Feihu, et al.
Publicado: (2026)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
por: Zhang, Yihua, et al.
Publicado: (2024)

Reading Between the Lines: Towards Reliable Black-box LLM Fingerprinting via Zeroth-order Gradient Estimation
por: Shao, Shuo, et al.
Publicado: (2025)

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
por: Sun, Yan, et al.
Publicado: (2025)

Data-efficient Fine-tuning for LLM-based Recommendation
por: Lin, Xinyu, et al.
Publicado: (2024)

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
por: Wang, Fei, et al.
Publicado: (2024)

Efficient Knowledge Graph Unlearning with Zeroth-order Information
por: Xiao, Yang, et al.
Publicado: (2025)

DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
por: Wang, Liangyu, et al.
Publicado: (2025)

Variable-order fractional Laplacian and its accurate and efficient computations with meshfree methods
por: Wu, Yixuan, et al.
Publicado: (2024)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
por: Cai, Zhijie, et al.
Publicado: (2026)

Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMs
por: Tan, Qitao, et al.
Publicado: (2026)

LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation
por: Xiao, Yang, et al.
Publicado: (2025)

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
por: Zhou, Huichi, et al.
Publicado: (2025)

EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction
por: Lin, Hsi-Che, et al.
Publicado: (2025)

Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
por: Yao, Yongqiang, et al.
Publicado: (2025)

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models
por: Yao, Kai, et al.
Publicado: (2024)

Alleviating the Fear of Losing Alignment in LLM Fine-tuning
por: Yang, Kang, et al.
Publicado: (2025)

Fine-tuning CLIP Text Encoders with Two-step Paraphrasing
por: Kim, Hyunjae, et al.
Publicado: (2024)

Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
por: Li, Zeman, et al.
Publicado: (2024)

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
por: Dang, Sizhe, et al.
Publicado: (2025)

Zeroth-order Logconcave Sampling
por: Kook, Yunbum, et al.
Publicado: (2025)

Zeroth-order parallel sampling
por: Pozza, Francesco, et al.
Publicado: (2026)

Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks
por: Youstra, Jack, et al.
Publicado: (2025)

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
por: Fan, Jiajun, et al.
Publicado: (2025)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
por: Yang, Yifan, et al.
Publicado: (2024)