Cover Image

ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Liangyu, Ren, Jie, Xu, Hang, Wang, Junxiao, Xie, Huanyi, Keyes, David E., Wang, Di
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Performance
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing
by: Wang, Liangyu, et al.
Published: (2025)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
by: Zhou, Jiajun, et al.
Published: (2025)

GPU-Accelerated Parallel Selected Inversion for Structured Matrices Using sTiles
by: Fattah, Esmail Abdul, et al.
Published: (2025)

FlashDP: Private Training Large Language Models with Efficient DP-SGD
by: Wang, Liangyu, et al.
Published: (2025)

ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
by: Peng, Yuezhang, et al.
Published: (2025)

MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models
by: Zhang, Zhen, et al.
Published: (2025)

Tuning Fast Memory Size based on Modeling of Page Migration for Tiered Memory
by: Chen, Shangye, et al.
Published: (2024)

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
by: Wang, Shuo, et al.
Published: (2026)

Heterogeneous Memory Pool Tuning
by: Vaverka, Filip, et al.
Published: (2025)

WaveTune: Wave-aware Bilinear Modeling for Efficient GPU Kernel Auto-tuning
by: Zhang, Kaixuan, et al.
Published: (2026)

DSO: A GPU Energy Efficiency Optimizer by Fusing Dynamic and Static Information
by: Wang, Qiang, et al.
Published: (2024)

Machine Learning-Guided Memory Optimization for DLRM Inference on Tiered Memory
by: Ren, Jie, et al.
Published: (2025)

Scalable GPU Performance Variability Analysis framework
by: Lahiry, Ankur, et al.
Published: (2025)

CUTHERMO: Understanding GPU Memory Inefficiencies with Heat Map Profiling
by: Zhao, Yanbo, et al.
Published: (2025)

Rotary GPU: Exploring Local Execution Paths for Large Mixture-of-Experts Models Under Limited GPU Memory
by: Jo, Myeong Jun
Published: (2026)

AGIPC: Adaptive In-Solve Algebraic Coarsening for GPU IPC
by: Wang, Xuan, et al.
Published: (2026)

sTiles: An Accelerated Computational Framework for Sparse Factorizations of Structured Matrices
by: Fattah, Esmail Abdul, et al.
Published: (2025)

A Precision Emulation Approach to the GPU Acceleration of Ab Initio Electronic Structure Calculations
by: Liu, Hang, et al.
Published: (2026)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
by: Liu, Yong, et al.
Published: (2024)

GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
by: Taneja, Maanas, et al.
Published: (2026)

The Theoretical Limit of Radar Target Detection
by: Xu, Dazhuan, et al.
Published: (2021)

ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
by: Zhao, Youpeng, et al.
Published: (2024)

KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
by: Wang, Han, et al.
Published: (2026)

A Limits Study of Memory-side Tiering Telemetry
by: Petrucci, Vinicius, et al.
Published: (2025)

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
by: Guo, Wentao, et al.
Published: (2024)

ALISE: Accelerating Large Language Model Serving with Speculative Scheduling
by: Zhao, Youpeng, et al.
Published: (2024)

GPUVM: GPU-driven Unified Virtual Memory
by: Nazaraliyev, Nurlan, et al.
Published: (2024)

Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models
by: Wang, Fei, et al.
Published: (2024)

SemaTune: Semantic-Aware Online OS Tuning with Large Language Models
by: Liargkovas, Georgios, et al.
Published: (2026)

LLMPerf: GPU Performance Modeling meets Large Language Models
by: Nguyen, Khoi N. M., et al.
Published: (2025)

Delegation with Trust<T>: A Scalable, Type- and Memory-Safe Alternative to Locks
by: Ahmad, Noaman, et al.
Published: (2024)

AMD MI300X GPU Performance Analysis
by: Ambati, Chandrish, et al.
Published: (2025)

A Quantitative Analysis and Guidelines of Data Streaming Accelerator in Modern Intel Xeon Scalable Processors
by: Kuper, Reese, et al.
Published: (2023)

Fast and Scalable Mixed Precision Euclidean Distance Calculations Using GPU Tensor Cores
by: Curless, Brian, et al.
Published: (2025)

Conformer-Based Speech Recognition On Extreme Edge-Computing Devices
by: Xu, Mingbin, et al.
Published: (2023)

L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning
by: Singh, Raul, et al.
Published: (2025)

Cloud Computing Energy Consumption Prediction Based on Kernel Extreme Learning Machine Algorithm Improved by Vector Weighted Average Algorithm
by: Wang, Yuqing, et al.
Published: (2025)

Dissecting CPU-GPU Unified Physical Memory on AMD MI300A APUs
by: Wahlgren, Jacob, et al.
Published: (2025)

gDist: Efficient Distance Computation between 3D Meshes on GPU
by: Fang, Peng, et al.
Published: (2024)

You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models
by: Ding, Shiwei, et al.
Published: (2025)