Cover Image

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Guo, Wentao, Long, Jikai, Zeng, Yimeng, Liu, Zirui, Yang, Xinyu, Ran, Yide, Gardner, Jacob R., Bastani, Osbert, De Sa, Christopher, Yu, Xiaodong, Chen, Beidi, Xu, Zhaozhuo
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
by: Ran, Yide, et al.
Published: (2025)

OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning
by: Long, Jikai, et al.
Published: (2025)

Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting
by: Young, Halley, et al.
Published: (2024)

Generative Adversarial Model-Based Optimization via Source Critic Regularization
by: Yao, Michael S., et al.
Published: (2024)

Sirius: Contextual Sparsity with Correction for Efficient LLMs
by: Zhou, Yang, et al.
Published: (2024)

Asymptotic Normality of Generalized Low-Rank Matrix Sensing via Riemannian Geometry
by: Bastani, Osbert
Published: (2024)

Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
by: Wu, Yuheng, et al.
Published: (2025)

Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
by: Yu, Ziming, et al.
Published: (2024)

Adversarial Query Synthesis via Bayesian Optimization
by: Tao, Jeffrey, et al.
Published: (2026)

SPARLING: Learning Latent Representations with Extremely Sparse Activations
by: Gupta, Kavi, et al.
Published: (2023)

Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
by: Zhang, Kairun, et al.
Published: (2025)

Stochastic Online Conformal Prediction with Semi-Bandit Feedback
by: Ge, Haosen, et al.
Published: (2024)

Are AI Capabilities Increasing Exponentially? A Competing Hypothesis
by: Ge, Haosen, et al.
Published: (2026)

Rethinking Algorithmic Fairness for Human-AI Collaboration
by: Ge, Haosen, et al.
Published: (2023)

Beating the Winner's Curse via Inference-Aware Policy Optimization
by: Bastani, Hamsa, et al.
Published: (2025)

Improving Human Sequential Decision-Making with Reinforcement Learning
by: Bastani, Hamsa, et al.
Published: (2021)

Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching
by: Bastani, Hamsa, et al.
Published: (2026)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
by: Liu, Yong, et al.
Published: (2024)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
by: Zhang, Yihua, et al.
Published: (2024)

LLM Zeroth-Order Fine-Tuning is an Inference Workload
by: Li, Zelin, et al.
Published: (2026)

Synthesizing Trajectory Queries from Examples
by: Mell, Stephen, et al.
Published: (2026)

Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings
by: Xu, Kan, et al.
Published: (2021)

Stochastic Bandits with ReLU Neural Networks
by: Xu, Kan, et al.
Published: (2024)

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
by: Zhao, Yanjun, et al.
Published: (2024)

Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
by: Mi, Zhendong, et al.
Published: (2025)

Learning Performance-Improving Code Edits
by: Shypula, Alexander, et al.
Published: (2023)

KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning
by: Mi, Zhendong, et al.
Published: (2025)

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
by: Liu, Zirui, et al.
Published: (2024)

ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
by: Wang, Liangyu, et al.
Published: (2025)

Token-wise Influential Training Data Retrieval for Large Language Models
by: Lin, Huawei, et al.
Published: (2024)

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
by: Si, Wenwen, et al.
Published: (2026)

Decaf: Improving Neural Decompilation with Automatic Feedback and Search
by: Shypula, Alexander, et al.
Published: (2026)

Conformal Structured Prediction
by: Zhang, Botong, et al.
Published: (2024)

Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction
by: Ramalingam, Ramya, et al.
Published: (2024)

LLM Program Optimization via Retrieval Augmented Search
by: Anupam, Sagnik, et al.
Published: (2025)

Prior-Agnostic Incentive-Compatible Exploration
by: Ramalingam, Ramya, et al.
Published: (2026)

Optimal Program Synthesis via Abstract Interpretation
by: Mell, Stephen, et al.
Published: (2026)

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
by: Dang, Sizhe, et al.
Published: (2025)

On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
by: Katti, Prabodh, et al.
Published: (2025)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
by: Gautam, Tanmay, et al.
Published: (2024)