Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2503.11314 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915335999848448 |
|---|---|
| author | Tang, Xinyu Wang, Xiaolei Lv, Zhihao Min, Yingqian Zhao, Wayne Xin Hu, Binbin Liu, Ziqi Zhang, Zhiqiang |
| author_facet | Tang, Xinyu Wang, Xiaolei Lv, Zhihao Min, Yingqian Zhao, Wayne Xin Hu, Binbin Liu, Ziqi Zhang, Zhiqiang |
| contents | Recent advancements in long chain-of-thoughts(long CoTs) have significantly improved the reasoning capabilities of large language models(LLMs). Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples and can easily transfer to other tasks. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. In this work, we conduct an empirical analysis for this question from the perspective of representation. We find that LLMs do encode long CoT reasoning as a general capability, with a clear distinction from vanilla CoTs. Furthermore, domain-specific representations are also required for the effective transfer of long CoT reasoning. Inspired by these findings, we propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs. Extensive experiments demonstrate the effectiveness and efficiency of GLoRE in both in-domain and cross-domain scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11314 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering Tang, Xinyu Wang, Xiaolei Lv, Zhihao Min, Yingqian Zhao, Wayne Xin Hu, Binbin Liu, Ziqi Zhang, Zhiqiang Computation and Language Recent advancements in long chain-of-thoughts(long CoTs) have significantly improved the reasoning capabilities of large language models(LLMs). Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples and can easily transfer to other tasks. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. In this work, we conduct an empirical analysis for this question from the perspective of representation. We find that LLMs do encode long CoT reasoning as a general capability, with a clear distinction from vanilla CoTs. Furthermore, domain-specific representations are also required for the effective transfer of long CoT reasoning. Inspired by these findings, we propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs. Extensive experiments demonstrate the effectiveness and efficiency of GLoRE in both in-domain and cross-domain scenarios. |
| title | Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.11314 |