Cover Image

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Yong, Zhu, Zirui, Gong, Chaoyu, Cheng, Minhao, Hsieh, Cho-Jui, You, Yang
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
by: Wang, Shuo, et al.
Published: (2026)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
by: Cai, Zhijie, et al.
Published: (2026)

DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers
by: Li, Xirui, et al.
Published: (2024)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
by: Zhou, Jiajun, et al.
Published: (2025)

Two-stage LLM Fine-tuning with Less Specialization and More Generalization
by: Wang, Yihan, et al.
Published: (2022)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
by: Zhang, Yihua, et al.
Published: (2024)

Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection
by: Shen, Lingzhi, et al.
Published: (2025)

ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
by: Peng, Yuezhang, et al.
Published: (2025)

On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
by: Katti, Prabodh, et al.
Published: (2025)

Exploring Expert Failures Improves LLM Agent Tuning
by: Lan, Li-Cheng, et al.
Published: (2025)

Prior-Informed Zeroth-Order Optimization with Adaptive Direction Alignment for Memory-Efficient LLM Fine-Tuning
by: Jin, Feihu, et al.
Published: (2026)

MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models
by: Zhang, Zhen, et al.
Published: (2025)

POME: Post Optimization Model Edit via Muon-style Projection
by: Liu, Yong, et al.
Published: (2025)

Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
by: Jin, Feihu, et al.
Published: (2026)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
by: Gautam, Tanmay, et al.
Published: (2024)

Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
by: Li, Zeman, et al.
Published: (2024)

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models
by: Yao, Kai, et al.
Published: (2024)

ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
by: Wang, Liangyu, et al.
Published: (2025)

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
by: Wang, Ruochen, et al.
Published: (2024)

On the Loss of Context-awareness in General Instruction Fine-tuning
by: Wang, Yihan, et al.
Published: (2024)

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
by: Wang, Fei, et al.
Published: (2024)

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting
by: Gong, Xuchen, et al.
Published: (2025)

MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?
by: Li, Xirui, et al.
Published: (2024)

AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
by: Chen, Ziye, et al.
Published: (2026)

Hint Tuning: Less Data Makes Better Reasoners
by: Fan, Siqi, et al.
Published: (2026)

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
by: Ling, Zhenqing, et al.
Published: (2024)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
by: Li, Zelin, et al.
Published: (2026)

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
by: Guo, Wentao, et al.
Published: (2024)

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
by: Sun, Yan, et al.
Published: (2025)

Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer
by: Du, Guodong, et al.
Published: (2025)

AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization
by: Cemri, Mert, et al.
Published: (2026)

Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Model
by: Chih, Yu Cheng, et al.
Published: (2025)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
by: Yang, Yifan, et al.
Published: (2024)

MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusion
by: Li, Sen, et al.
Published: (2024)

Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
by: Prakriya, Neha, et al.
Published: (2024)

Pensez: Less Data, Better Reasoning -- Rethinking French LLM
by: Ha, Huy Hoang
Published: (2025)

The Harder The Better: Maintaining Supervised Fine-tuning Generalization with Less but Harder Data
by: Shang, Zhaoyang, et al.
Published: (2025)

On Discrete Prompt Optimization for Diffusion Models
by: Wang, Ruochen, et al.
Published: (2024)

From Parameters to Prompts: Understanding and Mitigating the Factuality Gap between Fine-Tuned LLMs
by: Gong, Xuan, et al.
Published: (2025)

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
by: Seung, Hyunseok, et al.
Published: (2025)