Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913097647652864 |
|---|---|
| author | Sonoda, Sho Hashimoto, Yuka Ishikawa, Isao Ikeda, Masahiro |
| author_facet | Sonoda, Sho Hashimoto, Yuka Ishikawa, Isao Ikeda, Masahiro |
| contents | Why and when does depth improve generalization? We study this question in an implementation-agnostic state-transition model, where a depth-$k$ predictor is a readout class $H$ composed with the word ball $B(k,F)$ generated by hidden state transitions. Generalization bounds separate implementation error, approximation error, and statistical complexity, and upper bound the depth-dependent variance term by a Dudley entropy integral over $B(k,F)$, with a conditional lower-bound diagnostic under readout separation. We identify geometric and semigroup mechanisms that keep this entropy contribution saturated or polynomial, and contrast them with separation mechanisms that recover the classical exponential-growth obstruction. Coupling these variance upper bounds with approximation rates gives typical depth trade-off patterns, clarifying that depth is statistically favorable when approximation improves rapidly while the transition semigroup remains geometrically tame. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15064 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning Sonoda, Sho Hashimoto, Yuka Ishikawa, Isao Ikeda, Masahiro Machine Learning Dynamical Systems Why and when does depth improve generalization? We study this question in an implementation-agnostic state-transition model, where a depth-$k$ predictor is a readout class $H$ composed with the word ball $B(k,F)$ generated by hidden state transitions. Generalization bounds separate implementation error, approximation error, and statistical complexity, and upper bound the depth-dependent variance term by a Dudley entropy integral over $B(k,F)$, with a conditional lower-bound diagnostic under readout separation. We identify geometric and semigroup mechanisms that keep this entropy contribution saturated or polynomial, and contrast them with separation mechanisms that recover the classical exponential-growth obstruction. Coupling these variance upper bounds with approximation rates gives typical depth trade-off patterns, clarifying that depth is statistically favorable when approximation improves rapidly while the transition semigroup remains geometrically tame. |
| title | Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning |
| topic | Machine Learning Dynamical Systems |
| url | https://arxiv.org/abs/2505.15064 |