Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917011465961472 |
|---|---|
| author | Urdshals, Einar Lau, Edmund Hoogland, Jesse van Wingerden, Stan Murfet, Daniel |
| author_facet | Urdshals, Einar Lau, Edmund Hoogland, Jesse van Wingerden, Stan Murfet, Daniel |
| contents | We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12077 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory Urdshals, Einar Lau, Edmund Hoogland, Jesse van Wingerden, Stan Murfet, Daniel Machine Learning We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression. |
| title | Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.12077 |