A Latent Variable Framework for Scaling Laws in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914185115336704 |
|---|---|
| author | Cai, Peiyao Cui, Chengyu Polo, Felipe Maia Somerstep, Seamus Choshen, Leshem Yurochkin, Mikhail Banerjee, Moulinath Sun, Yuekai Tan, Kean Ming Xu, Gongjun |
| author_facet | Cai, Peiyao Cui, Chengyu Polo, Felipe Maia Somerstep, Seamus Choshen, Leshem Yurochkin, Mikhail Banerjee, Moulinath Sun, Yuekai Tan, Kean Ming Xu, Gongjun |
| contents | We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new LLM families with distinct architectures and training strategies, evaluated on an increasing number of benchmarks. This heterogeneity makes a single global scaling curve inadequate for capturing how performance varies across families and benchmarks. To address this, we propose a latent variable modeling framework in which each LLM family is associated with a latent variable that captures the common underlying features in that family. An LLM's performance on different benchmarks is then driven by its latent skills, which are jointly determined by the latent variable and the model's own observable features. We develop an estimation procedure for this latent variable model and establish its statistical properties. We also design efficient numerical algorithms that support estimation and various downstream tasks. Empirically, we evaluate the approach on 12 widely used benchmarks from the Open LLM Leaderboard (v1/v2). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_06553 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Latent Variable Framework for Scaling Laws in Large Language Models Cai, Peiyao Cui, Chengyu Polo, Felipe Maia Somerstep, Seamus Choshen, Leshem Yurochkin, Mikhail Banerjee, Moulinath Sun, Yuekai Tan, Kean Ming Xu, Gongjun Applications Machine Learning We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new LLM families with distinct architectures and training strategies, evaluated on an increasing number of benchmarks. This heterogeneity makes a single global scaling curve inadequate for capturing how performance varies across families and benchmarks. To address this, we propose a latent variable modeling framework in which each LLM family is associated with a latent variable that captures the common underlying features in that family. An LLM's performance on different benchmarks is then driven by its latent skills, which are jointly determined by the latent variable and the model's own observable features. We develop an estimation procedure for this latent variable model and establish its statistical properties. We also design efficient numerical algorithms that support estimation and various downstream tasks. Empirically, we evaluate the approach on 12 widely used benchmarks from the Open LLM Leaderboard (v1/v2). |
| title | A Latent Variable Framework for Scaling Laws in Large Language Models |
| topic | Applications Machine Learning |
| url | https://arxiv.org/abs/2512.06553 |