Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908859342258176 |
|---|---|
| author | Nam, Yoonsoo Fonseca, Nayara Lee, Seok Hyeong Mingard, Chris Goring, Niclas Harzli, Ouns El Erturk, Abdurrahman Hadi Hayou, Soufiane Louis, Ard A. |
| author_facet | Nam, Yoonsoo Fonseca, Nayara Lee, Seok Hyeong Mingard, Chris Goring, Niclas Harzli, Ouns El Erturk, Abdurrahman Hadi Hayou, Soufiane Louis, Ard A. |
| contents | Dynamic feature transformation (the rich regime) does not always align with predictive performance (better representation), yet accuracy is often used as a proxy for richness, limiting analysis of their relationship. We propose a computationally efficient, performance-independent metric of richness grounded in the low-rank bias of rich dynamics, which recovers neural collapse as a special case. The metric is empirically more stable than existing alternatives and captures known lazy-torich transitions (e.g., grokking) without relying on accuracy. We further use it to examine how training factors (e.g., learning rate) relate to richness, confirming recognized assumptions and highlighting new observations (e.g., batch normalization promotes rich dynamics). An eigendecomposition-based visualization is also introduced to support interpretability, together providing a diagnostic tool for studying the relationship between training factors, dynamics, and representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04264 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement Nam, Yoonsoo Fonseca, Nayara Lee, Seok Hyeong Mingard, Chris Goring, Niclas Harzli, Ouns El Erturk, Abdurrahman Hadi Hayou, Soufiane Louis, Ard A. Machine Learning Dynamic feature transformation (the rich regime) does not always align with predictive performance (better representation), yet accuracy is often used as a proxy for richness, limiting analysis of their relationship. We propose a computationally efficient, performance-independent metric of richness grounded in the low-rank bias of rich dynamics, which recovers neural collapse as a special case. The metric is empirically more stable than existing alternatives and captures known lazy-torich transitions (e.g., grokking) without relying on accuracy. We further use it to examine how training factors (e.g., learning rate) relate to richness, confirming recognized assumptions and highlighting new observations (e.g., batch normalization promotes rich dynamics). An eigendecomposition-based visualization is also introduced to support interpretability, together providing a diagnostic tool for studying the relationship between training factors, dynamics, and representations. |
| title | Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.04264 |