Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910854621954048 |
|---|---|
| author | Chen, Yifang Guo, Xuyang Li, Xiaoyu Liang, Yingyu Shi, Zhenmei Song, Zhao |
| author_facet | Chen, Yifang Guo, Xuyang Li, Xiaoyu Liang, Yingyu Shi, Zhenmei Song, Zhao |
| contents | Recently, Large Language Models (LLMs) have achieved remarkable success. A key factor behind this success is the scaling law observed by OpenAI. Specifically, for models with Transformer architecture, the test loss exhibits a power-law relationship with model size, dataset size, and the amount of computation used in training, demonstrating trends that span more than seven orders of magnitude. This scaling law challenges traditional machine learning wisdom, notably the Oscar Scissors principle, which suggests that an overparametrized algorithm will overfit the training datasets, resulting in poor test performance. Recent research has also identified the scaling law in simpler machine learning contexts, such as linear regression. However, fully explaining the scaling law in large practical models remains an elusive goal. In this work, we advance our understanding by demonstrating that the scaling law phenomenon extends to multiple regression and kernel regression settings, which are significantly more expressive and powerful than linear methods. Our analysis provides deeper insights into the scaling law, potentially enhancing our understanding of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01314 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches Chen, Yifang Guo, Xuyang Li, Xiaoyu Liang, Yingyu Shi, Zhenmei Song, Zhao Machine Learning Artificial Intelligence Recently, Large Language Models (LLMs) have achieved remarkable success. A key factor behind this success is the scaling law observed by OpenAI. Specifically, for models with Transformer architecture, the test loss exhibits a power-law relationship with model size, dataset size, and the amount of computation used in training, demonstrating trends that span more than seven orders of magnitude. This scaling law challenges traditional machine learning wisdom, notably the Oscar Scissors principle, which suggests that an overparametrized algorithm will overfit the training datasets, resulting in poor test performance. Recent research has also identified the scaling law in simpler machine learning contexts, such as linear regression. However, fully explaining the scaling law in large practical models remains an elusive goal. In this work, we advance our understanding by demonstrating that the scaling law phenomenon extends to multiple regression and kernel regression settings, which are significantly more expressive and powerful than linear methods. Our analysis provides deeper insights into the scaling law, potentially enhancing our understanding of LLMs. |
| title | Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01314 |