LightThinker: Thinking Step-by-Step Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908553377218560 |
|---|---|
| author | Zhang, Jintian Zhu, Yuqi Sun, Mengshu Luo, Yujie Qiao, Shuofei Du, Lun Zheng, Da Chen, Huajun Zhang, Ningyu |
| author_facet | Zhang, Jintian Zhu, Yuqi Sun, Mengshu Luo, Yujie Qiao, Shuofei Du, Lun Zheng, Da Chen, Huajun Zhang, Ningyu |
| contents | Large language models (LLMs) have shown remarkable performance in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. In this paper, we propose LightThinker, a novel method that enables LLMs to dynamically compress intermediate thoughts during reasoning. Inspired by human cognitive processes, LightThinker compresses verbose thought steps into compact representations and discards the original reasoning chains, thereby significantly reducing the number of tokens stored in the context window. This is achieved by training the model on when and how to perform compression through data construction, mapping hidden states to condensed gist tokens, and creating specialized attention masks. Additionally, we introduce the Dependency (Dep) metric to quantify the degree of compression by measuring the reliance on historical tokens during generation. Extensive experiments on four datasets and two models show that LightThinker reduces peak memory usage and inference time, while maintaining competitive accuracy. Our work provides a new direction for improving the efficiency of LLMs in complex reasoning tasks without sacrificing performance. Code is released at https://github.com/zjunlp/LightThinker. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15589 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LightThinker: Thinking Step-by-Step Compression Zhang, Jintian Zhu, Yuqi Sun, Mengshu Luo, Yujie Qiao, Shuofei Du, Lun Zheng, Da Chen, Huajun Zhang, Ningyu Computation and Language Artificial Intelligence Information Retrieval Machine Learning Multimedia Large language models (LLMs) have shown remarkable performance in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. In this paper, we propose LightThinker, a novel method that enables LLMs to dynamically compress intermediate thoughts during reasoning. Inspired by human cognitive processes, LightThinker compresses verbose thought steps into compact representations and discards the original reasoning chains, thereby significantly reducing the number of tokens stored in the context window. This is achieved by training the model on when and how to perform compression through data construction, mapping hidden states to condensed gist tokens, and creating specialized attention masks. Additionally, we introduce the Dependency (Dep) metric to quantify the degree of compression by measuring the reliance on historical tokens during generation. Extensive experiments on four datasets and two models show that LightThinker reduces peak memory usage and inference time, while maintaining competitive accuracy. Our work provides a new direction for improving the efficiency of LLMs in complex reasoning tasks without sacrificing performance. Code is released at https://github.com/zjunlp/LightThinker. |
| title | LightThinker: Thinking Step-by-Step Compression |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning Multimedia |
| url | https://arxiv.org/abs/2502.15589 |