The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912244292386816 |
|---|---|
| author | Tang, Zhenheng Liu, Xiang Wang, Qian Dong, Peijie He, Bingsheng Chu, Xiaowen Li, Bo |
| author_facet | Tang, Zhenheng Liu, Xiang Wang, Qian Dong, Peijie He, Bingsheng Chu, Xiaowen Li, Bo |
| contents | Motivated by reducing the computational and storage costs of LLMs, model compression and KV cache compression have attracted much attention from researchers. However, current methods predominantly emphasize maintaining the performance of compressed LLMs, as measured by perplexity or simple accuracy on tasks of common sense knowledge QA and basic arithmetic reasoning. In this blog, we present a brief review of recent advancements in LLMs related to retrieval-augmented generation, multi-step reasoning, external tools, and computational expressivity, all of which substantially enhance LLM performance. Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and task, there exists a smaller lottery LLM capable of producing the same performance as the original LLM with the assistance of multi-step reasoning and external tools. Based on the review of current progress in LLMs, we discuss and summarize the essential capabilities that the lottery LLM and KV cache compression must possess, which are currently overlooked in existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17535 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve? Tang, Zhenheng Liu, Xiang Wang, Qian Dong, Peijie He, Bingsheng Chu, Xiaowen Li, Bo Machine Learning Artificial Intelligence Computation and Language Formal Languages and Automata Theory Motivated by reducing the computational and storage costs of LLMs, model compression and KV cache compression have attracted much attention from researchers. However, current methods predominantly emphasize maintaining the performance of compressed LLMs, as measured by perplexity or simple accuracy on tasks of common sense knowledge QA and basic arithmetic reasoning. In this blog, we present a brief review of recent advancements in LLMs related to retrieval-augmented generation, multi-step reasoning, external tools, and computational expressivity, all of which substantially enhance LLM performance. Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and task, there exists a smaller lottery LLM capable of producing the same performance as the original LLM with the assistance of multi-step reasoning and external tools. Based on the review of current progress in LLMs, we discuss and summarize the essential capabilities that the lottery LLM and KV cache compression must possess, which are currently overlooked in existing methods. |
| title | The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve? |
| topic | Machine Learning Artificial Intelligence Computation and Language Formal Languages and Automata Theory |
| url | https://arxiv.org/abs/2502.17535 |