Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations
Fuente:
arXiv
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918349273825280 |
|---|---|
| author | Ayena, Koffi O. |
| author_facet | Ayena, Koffi O. |
| contents | We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function $x^2$, with optimally chosen shift $a$ and scale $β$, we achieve approximation error $\varepsilon$ using a two-layer network of constant width, where weights scale as $β^{\pm k}$ with $k = \mathcal{O}(\ln(1/\varepsilon))$. We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth $\mathcal{O}(1)$ and $\mathcal{O}(\varepsilon^{-d/n})$ parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12132 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations Ayena, Koffi O. Machine Learning Numerical Analysis We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function $x^2$, with optimally chosen shift $a$ and scale $β$, we achieve approximation error $\varepsilon$ using a two-layer network of constant width, where weights scale as $β^{\pm k}$ with $k = \mathcal{O}(\ln(1/\varepsilon))$. We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth $\mathcal{O}(1)$ and $\mathcal{O}(\varepsilon^{-d/n})$ parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory. |
| title | Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2512.12132 |