Cover Image

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Huaqin, Li, Jiaxi, Pan, Yi, Liang, Shizhe, Yang, Xiaofeng, Liu, Wei, Li, Xiang, Dou, Fei, Liu, Tianming, Lu, Jin
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
by: Shang, Sifeng, et al.
Published: (2025)

Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
by: Wang, Fei, et al.
Published: (2026)

Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction
by: Liu, Zhexiong, et al.
Published: (2025)

Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
by: Mi, Zhendong, et al.
Published: (2025)

Private Fine-tuning of Large Language Models with Zeroth-order Optimization
by: Tang, Xinyu, et al.
Published: (2024)

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
by: Tan, Qitao, et al.
Published: (2025)

GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning
by: Tian, Kaiyuan, et al.
Published: (2026)

Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping
by: Pan, Zijie, et al.
Published: (2023)

Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimization
by: Li, Zhe, et al.
Published: (2025)

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models
by: Yao, Kai, et al.
Published: (2024)

LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
by: Roh, Yuji, et al.
Published: (2024)

Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
by: Chen, Yiming, et al.
Published: (2024)

ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
by: Meng, Chuiyang, et al.
Published: (2026)

Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
by: Sadiev, Abdurakhmon, et al.
Published: (2025)

Zeroth-order Low-rank Hessian Estimation via Matrix Recovery
by: Wang, Tianyu, et al.
Published: (2024)

Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning
by: You, Weihang, et al.
Published: (2026)

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
by: Zhao, Yanjun, et al.
Published: (2024)

Subspace-based Approximate Hessian Method for Zeroth-Order Optimization
by: Kim, Dongyoon, et al.
Published: (2025)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
by: Cai, Zhijie, et al.
Published: (2026)

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
by: Sun, Yan, et al.
Published: (2025)

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
by: Wang, Fei, et al.
Published: (2024)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
by: Zhang, Yihua, et al.
Published: (2024)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
by: Li, Zelin, et al.
Published: (2026)

Three Birds, One Stone: Solving the Communication-Memory-Privacy Trilemma in LLM Fine-tuning Over Wireless Networks with Zeroth-Order Optimization
by: Cai, Zhijie, et al.
Published: (2026)

Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis
by: Xu, Shaochen, et al.
Published: (2024)

KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning
by: Mi, Zhendong, et al.
Published: (2025)

Survey of HPC in US Research Institutions
by: Shu, Peng, et al.
Published: (2025)

DCPO: Dynamic Clipping Policy Optimization
by: Yang, Shihui, et al.
Published: (2025)

Optimized Gradient Clipping for Noisy Label Learning
by: Ye, Xichen, et al.
Published: (2024)

An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization
by: Liu, Shanglin, et al.
Published: (2024)

Accelerating Single-Point Zeroth-Order Optimization with Regression-Based Gradient Surrogates
by: Chen, Xin, et al.
Published: (2025)

Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillation
by: Peng, Tianhao, et al.
Published: (2023)

Helen: Optimizing CTR Prediction Models with Frequency-wise Hessian Eigenvalue Regularization
by: Zhu, Zirui, et al.
Published: (2024)

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization
by: Marrinan, Luke, et al.
Published: (2023)

Median Clipping for Zeroth-order Non-Smooth Convex Optimization and Multi-Armed Bandit Problem with Heavy-tailed Symmetric Noise
by: Kornilov, Nikita, et al.
Published: (2024)

Riemannian Accelerated Zeroth-order Algorithm: Improved Robustness and Lower Query Complexity
by: He, Chang, et al.
Published: (2024)

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
by: Seung, Hyunseok, et al.
Published: (2025)

Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models
by: Liu, Ziwei, et al.
Published: (2025)

Reading Between the Lines: Towards Reliable Black-box LLM Fingerprinting via Zeroth-order Gradient Estimation
by: Shao, Shuo, et al.
Published: (2025)

Outlier-weighed Layerwise Sampling for LLM Fine-tuning
by: Li, Pengxiang, et al.
Published: (2024)