Cover Image

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Legate, Gwen, Rish, Irina, Belilovsky, Eugene
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
by: Roger, Alexis, et al.
Published: (2025)

LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
by: Roger, Alexis, et al.
Published: (2026)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
by: Ibrahim, Adam, et al.
Published: (2024)

Continual Pre-training of MoEs: How robust is your router?
by: Thérien, Benjamin, et al.
Published: (2025)

Celo2: Towards Learned Optimization Free Lunch
by: Moudgil, Abhinav, et al.
Published: (2026)

Efficient Refusal Ablation in LLM through Optimal Transport
by: Nanfack, Geraldin, et al.
Published: (2026)

Accelerating Training with Neuron Interaction and Nowcasting Networks
by: Knyazev, Boris, et al.
Published: (2024)

Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
by: Hameed, Humza Wajid, et al.
Published: (2024)

A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learning
by: Wu, Chenlin, et al.
Published: (2024)

ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
by: Nabli, Adel, et al.
Published: (2024)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
by: Abbes, Istabrak, et al.
Published: (2025)

Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
by: Ran, Yide, et al.
Published: (2025)

Refining Adaptive Zeroth-Order Optimization at Ease
by: Shu, Yao, et al.
Published: (2025)

Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
by: Riemer, Matthew, et al.
Published: (2024)

OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning
by: Long, Jikai, et al.
Published: (2025)

Less is More: Undertraining Experts Improves Model Upcycling
by: Horoi, Stefan, et al.
Published: (2025)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
by: Thérien, Benjamin, et al.
Published: (2025)

Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
by: Yu, Ziming, et al.
Published: (2024)

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
by: Guo, Wentao, et al.
Published: (2024)

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
by: Singh, Vaibhav, et al.
Published: (2025)

Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training
by: Cunegatti, Elia, et al.
Published: (2024)

Model Parallelism With Subnetwork Data Parallelism
by: Singh, Vaibhav, et al.
Published: (2025)

Non-Uniform Parameter-Wise Model Merging
by: Camacho, Albert Manuel Orozco, et al.
Published: (2024)

When Foresight Pruning Meets Zeroth-Order Optimization: Efficient Federated Learning for Low-Memory Devices
by: Zhang, Pengyu, et al.
Published: (2024)

Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
by: Xiao, Mingqing, et al.
Published: (2024)

Subspace-based Approximate Hessian Method for Zeroth-Order Optimization
by: Kim, Dongyoon, et al.
Published: (2025)

Heterogeneous Low-Bandwidth Pre-Training of LLMs
by: Obeidi, Yazan, et al.
Published: (2026)

DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
by: Singh, Vaibhav, et al.
Published: (2025)

Zeroth-Order Optimization Finds Flat Minima
by: Zhang, Liang, et al.
Published: (2025)

Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting
by: Riachi, Roland, et al.
Published: (2025)

HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
by: Chen, Qiyuan, et al.
Published: (2026)

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
by: Zhou, Jiajun, et al.
Published: (2025)

Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training
by: Tan, Qitao, et al.
Published: (2025)

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles
by: Tang, Zhiwei, et al.
Published: (2023)

Federated Linear Contextual Bandits with Heterogeneous Clients
by: Blaser, Ethan, et al.
Published: (2024)

On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization
by: Ma, Shaocong, et al.
Published: (2025)

Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizer
by: Feng, Jian, et al.
Published: (2026)

ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
by: Meng, Chuiyang, et al.
Published: (2026)

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
by: Jain, Arnav Kumar, et al.
Published: (2024)

Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
by: Mircea, Andrei, et al.
Published: (2025)