Mapping 1,000+ Language Models via the Log-Likelihood Vector
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908387502981120 |
|---|---|
| author | Oyama, Momose Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi |
| author_facet | Oyama, Momose Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi |
| contents | To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities. Our method is highly scalable, with computational cost growing linearly in both the number of models and text samples, and is easy to implement as the required features are derived from cross-entropy loss. Applying this method to over 1,000 language models, we constructed a "model map," providing a new perspective on large-scale model analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16173 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mapping 1,000+ Language Models via the Log-Likelihood Vector Oyama, Momose Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi Computation and Language To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities. Our method is highly scalable, with computational cost growing linearly in both the number of models and text samples, and is easy to implement as the required features are derived from cross-entropy loss. Applying this method to over 1,000 language models, we constructed a "model map," providing a new perspective on large-scale model analysis. |
| title | Mapping 1,000+ Language Models via the Log-Likelihood Vector |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.16173 |