Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909053859397632 |
|---|---|
| author | Liu, Sijia Lang, Yicheng Pal, Soumyadeep Wang, Changsheng Huang, Yancheng Fan, Chongyu Diffenderfer, James Kailkhura, Bhavya Zhang, Yihua |
| author_facet | Liu, Sijia Lang, Yicheng Pal, Soumyadeep Wang, Changsheng Huang, Yancheng Fan, Chongyu Diffenderfer, James Kailkhura, Bhavya Zhang, Yihua |
| contents | Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15622 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered Liu, Sijia Lang, Yicheng Pal, Soumyadeep Wang, Changsheng Huang, Yancheng Fan, Chongyu Diffenderfer, James Kailkhura, Bhavya Zhang, Yihua Machine Learning Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization. |
| title | Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.15622 |