Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.25074 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917240175067136 |
|---|---|
| author | Clark, Andrew Moursounidis, Jack Rasouli, Osmaan Gan, William Doyle, Cooper Leontjeva, Anna |
| author_facet | Clark, Andrew Moursounidis, Jack Rasouli, Osmaan Gan, William Doyle, Cooper Leontjeva, Anna |
| contents | We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating the gradients of black-box functions. BOND is distinguished by its formulation, which adaptively bounds perturbations to ensure accurate sign estimation, and by its implementation, which operates at black-box interfaces. This enables BOND to be more accurate and scalable compared to existing methods, facilitating end-to-end training of architectures that incorporate non-autodifferentiable modules. We observe that these modules, implemented in our experiments as frozen networks, can enhance model performance without increasing the number of trainable parameters. Our findings highlight the potential of leveraging fixed transformations to expand model capacity, pointing to hybrid analogue - digital devices as a path to scaling networks, and provides insights into the dynamics of adaptive optimizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25074 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BOND: License to Train with Black-Box Functions Clark, Andrew Moursounidis, Jack Rasouli, Osmaan Gan, William Doyle, Cooper Leontjeva, Anna Machine Learning I.2.6 We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating the gradients of black-box functions. BOND is distinguished by its formulation, which adaptively bounds perturbations to ensure accurate sign estimation, and by its implementation, which operates at black-box interfaces. This enables BOND to be more accurate and scalable compared to existing methods, facilitating end-to-end training of architectures that incorporate non-autodifferentiable modules. We observe that these modules, implemented in our experiments as frozen networks, can enhance model performance without increasing the number of trainable parameters. Our findings highlight the potential of leveraging fixed transformations to expand model capacity, pointing to hybrid analogue - digital devices as a path to scaling networks, and provides insights into the dynamics of adaptive optimizers. |
| title | BOND: License to Train with Black-Box Functions |
| topic | Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2510.25074 |