Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.03944 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917858184790016 |
|---|---|
| author | Yang, Hao Zhao, Qianghua Li, Lei |
| author_facet | Yang, Hao Zhao, Qianghua Li, Lei |
| contents | Chain-of-Thought prompting has significantly enhanced the reasoning capabilities of large language models, with numerous studies exploring factors influencing its performance. However, the underlying mechanisms remain poorly understood. To further demystify the operational principles, this work examines three key aspects: decoding, projection, and activation, aiming to elucidate the changes that occur within models when employing Chainof-Thought. Our findings reveal that LLMs effectively imitate exemplar formats while integrating them with their understanding of the question, exhibiting fluctuations in token logits during generation but ultimately producing a more concentrated logits distribution, and activating a broader set of neurons in the final layers, indicating more extensive knowledge retrieval compared to standard prompts. Our code and data will be publicly avialable when the paper is accepted. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03944 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation Yang, Hao Zhao, Qianghua Li, Lei Artificial Intelligence Chain-of-Thought prompting has significantly enhanced the reasoning capabilities of large language models, with numerous studies exploring factors influencing its performance. However, the underlying mechanisms remain poorly understood. To further demystify the operational principles, this work examines three key aspects: decoding, projection, and activation, aiming to elucidate the changes that occur within models when employing Chainof-Thought. Our findings reveal that LLMs effectively imitate exemplar formats while integrating them with their understanding of the question, exhibiting fluctuations in token logits during generation but ultimately producing a more concentrated logits distribution, and activating a broader set of neurons in the final layers, indicating more extensive knowledge retrieval compared to standard prompts. Our code and data will be publicly avialable when the paper is accepted. |
| title | Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2412.03944 |