Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.14171 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909228099174400 |
|---|---|
| author | Guo, Peijia Li, Ziguang Hu, Haibo Huang, Chao Li, Ming Zhang, Rui |
| author_facet | Guo, Peijia Li, Ziguang Hu, Haibo Huang, Chao Li, Ming Zhang, Rui |
| contents | We conceptualize the process of understanding as information compression, and propose a method for ranking large language models (LLMs) based on lossless data compression. We demonstrate the equivalence of compression length under arithmetic coding with cumulative negative log probabilities when using a large language model as a prior, that is, the pre-training phase of the model is essentially the process of learning the optimal coding length. At the same time, the evaluation metric compression ratio can be obtained without actual compression, which greatly saves overhead. In this paper, we use five large language models as priors for compression, then compare their performance on challenging natural language processing tasks, including sentence completion, question answering, and coreference resolution. Experimental results show that compression ratio and model performance are positively correlated, so it can be used as a general metric to evaluate large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14171 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ranking LLMs by compression Guo, Peijia Li, Ziguang Hu, Haibo Huang, Chao Li, Ming Zhang, Rui Artificial Intelligence Computation and Language We conceptualize the process of understanding as information compression, and propose a method for ranking large language models (LLMs) based on lossless data compression. We demonstrate the equivalence of compression length under arithmetic coding with cumulative negative log probabilities when using a large language model as a prior, that is, the pre-training phase of the model is essentially the process of learning the optimal coding length. At the same time, the evaluation metric compression ratio can be obtained without actual compression, which greatly saves overhead. In this paper, we use five large language models as priors for compression, then compare their performance on challenging natural language processing tasks, including sentence completion, question answering, and coreference resolution. Experimental results show that compression ratio and model performance are positively correlated, so it can be used as a general metric to evaluate large language models. |
| title | Ranking LLMs by compression |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.14171 |