PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908697877282816 |
|---|---|
| author | Yax, Nicolas Oudeyer, Pierre-Yves Palminteri, Stefano |
| author_facet | Yax, Nicolas Oudeyer, Pierre-Yves Palminteri, Stefano |
| contents | This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks Yax, Nicolas Oudeyer, Pierre-Yves Palminteri, Stefano Computation and Language Machine Learning Populations and Evolution This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information. |
| title | PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks |
| topic | Computation and Language Machine Learning Populations and Evolution |
| url | https://arxiv.org/abs/2404.04671 |