Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.12918 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908877305413632 |
|---|---|
| author | Wen, Pengcheng Ji, Jiaming Chan, Chi-Min Dai, Juntao Hong, Donghai Yang, Yaodong Han, Sirui Guo, Yike |
| author_facet | Wen, Pengcheng Ji, Jiaming Chan, Chi-Min Dai, Juntao Hong, Donghai Yang, Yaodong Han, Sirui Guo, Yike |
| contents | Large language models (LLMs) have demonstrated enhanced performance through the \textit{Thinking then Responding} paradigm, where models generate internal thoughts before final responses (aka, System 2 thinking). However, existing research lacks a systematic understanding of the mechanisms underlying how thinking patterns affect performance across model sizes. In this work, we conduct a comprehensive analysis of the impact of various thinking types on model performance and introduce ThinkPatterns-21k, a curated dataset comprising 21k instruction-response pairs (QA) collected from existing instruction-following datasets with five thinking types. For each pair, we augment it with five distinct internal thinking patterns: one unstructured thinking (monologue) and four structured variants (decomposition, self-ask, self-debate and self-critic), while maintaining the same instruction and response. Through extensive evaluation across different model sizes (3B-32B parameters), we have two key findings: (1) smaller models (<30B parameters) can benefit from most of structured thinking patterns, while larger models (32B) with structured thinking like decomposition would degrade performance and (2) unstructured monologue demonstrates broad effectiveness across different model sizes. Finally, we released all of our datasets, checkpoints, training logs of diverse thinking patterns to reproducibility, aiming to facilitate further research in this direction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12918 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMs Wen, Pengcheng Ji, Jiaming Chan, Chi-Min Dai, Juntao Hong, Donghai Yang, Yaodong Han, Sirui Guo, Yike Computation and Language Large language models (LLMs) have demonstrated enhanced performance through the \textit{Thinking then Responding} paradigm, where models generate internal thoughts before final responses (aka, System 2 thinking). However, existing research lacks a systematic understanding of the mechanisms underlying how thinking patterns affect performance across model sizes. In this work, we conduct a comprehensive analysis of the impact of various thinking types on model performance and introduce ThinkPatterns-21k, a curated dataset comprising 21k instruction-response pairs (QA) collected from existing instruction-following datasets with five thinking types. For each pair, we augment it with five distinct internal thinking patterns: one unstructured thinking (monologue) and four structured variants (decomposition, self-ask, self-debate and self-critic), while maintaining the same instruction and response. Through extensive evaluation across different model sizes (3B-32B parameters), we have two key findings: (1) smaller models (<30B parameters) can benefit from most of structured thinking patterns, while larger models (32B) with structured thinking like decomposition would degrade performance and (2) unstructured monologue demonstrates broad effectiveness across different model sizes. Finally, we released all of our datasets, checkpoints, training logs of diverse thinking patterns to reproducibility, aiming to facilitate further research in this direction. |
| title | ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.12918 |