Understanding Generalization from Embedding Dimension and Distributional Convergence
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910006021980160 |
|---|---|
| author | Yu, Junjie Ouyang, Zhuoli Deng, Haotian Wei, Chen Ma, Wenxiao Zhang, Jianyu Deng, Zihan Liu, Quanying |
| author_facet | Yu, Junjie Ouyang, Zhuoli Deng, Haotian Wei, Chen Ma, Wenxiao Zhang, Jianyu Deng, Zihan Liu, Quanying |
| contents | Deep neural networks often generalize well despite heavy over-parameterization, challenging classical parameter-based analyses. We study generalization from a representation-centric perspective and analyze how the geometry of learned embeddings controls predictive performance for a fixed trained model. We show that population risk can be bounded by two factors: (i) the intrinsic dimension of the embedding distribution, which determines the convergence rate of empirical embedding distribution to the population distribution in Wasserstein distance, and (ii) the sensitivity of the downstream mapping from embeddings to predictions, characterized by Lipschitz constants. Together, these yield an embedding-dependent error bound that does not rely on parameter counts or hypothesis class complexity. At the final embedding layer, architectural sensitivity vanishes and the bound is dominated by embedding dimension, explaining its strong empirical correlation with generalization performance. Experiments across architectures and datasets validate the theory and demonstrate the utility of embedding-based diagnostics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22756 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Understanding Generalization from Embedding Dimension and Distributional Convergence Yu, Junjie Ouyang, Zhuoli Deng, Haotian Wei, Chen Ma, Wenxiao Zhang, Jianyu Deng, Zihan Liu, Quanying Machine Learning Deep neural networks often generalize well despite heavy over-parameterization, challenging classical parameter-based analyses. We study generalization from a representation-centric perspective and analyze how the geometry of learned embeddings controls predictive performance for a fixed trained model. We show that population risk can be bounded by two factors: (i) the intrinsic dimension of the embedding distribution, which determines the convergence rate of empirical embedding distribution to the population distribution in Wasserstein distance, and (ii) the sensitivity of the downstream mapping from embeddings to predictions, characterized by Lipschitz constants. Together, these yield an embedding-dependent error bound that does not rely on parameter counts or hypothesis class complexity. At the final embedding layer, architectural sensitivity vanishes and the bound is dominated by embedding dimension, explaining its strong empirical correlation with generalization performance. Experiments across architectures and datasets validate the theory and demonstrate the utility of embedding-based diagnostics. |
| title | Understanding Generalization from Embedding Dimension and Distributional Convergence |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.22756 |