Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914490551894016 |
|---|---|
| author | Chen, Beiduo Hu, Tiancheng Zhang, Caiqi Litschko, Robert Korhonen, Anna Plank, Barbara |
| author_facet | Chen, Beiduo Hu, Tiancheng Zhang, Caiqi Litschko, Robert Korhonen, Anna Plank, Barbara |
| contents | Reasoning-tuned LLMs utilizing long Chain-of-Thought (CoT) excel at single-answer tasks, yet their ability to model Human Label Variation--which requires capturing probabilistic ambiguity rather than resolving it--remains underexplored. We investigate this through systematic disentanglement experiments on distribution-based tasks, employing Cross-CoT experiments to isolate the effect of reasoning text from intrinsic model priors. We observe a distinct "decoupled mechanism": while CoT improves distributional alignment, final accuracy is dictated by CoT content (99% variance contribution), whereas distributional ranking is governed by model priors (over 80%). Step-wise analysis further shows that while CoT's influence on accuracy grows monotonically during the reasoning process, distributional structure is largely determined by LLM's intrinsic priors. These findings suggest that long CoT serves as a decisive LLM decision-maker for the top option but fails to function as a granular distribution calibrator for ambiguous tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03154 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective Chen, Beiduo Hu, Tiancheng Zhang, Caiqi Litschko, Robert Korhonen, Anna Plank, Barbara Computation and Language Reasoning-tuned LLMs utilizing long Chain-of-Thought (CoT) excel at single-answer tasks, yet their ability to model Human Label Variation--which requires capturing probabilistic ambiguity rather than resolving it--remains underexplored. We investigate this through systematic disentanglement experiments on distribution-based tasks, employing Cross-CoT experiments to isolate the effect of reasoning text from intrinsic model priors. We observe a distinct "decoupled mechanism": while CoT improves distributional alignment, final accuracy is dictated by CoT content (99% variance contribution), whereas distributional ranking is governed by model priors (over 80%). Step-wise analysis further shows that while CoT's influence on accuracy grows monotonically during the reasoning process, distributional structure is largely determined by LLM's intrinsic priors. These findings suggest that long CoT serves as a decisive LLM decision-maker for the top option but fails to function as a granular distribution calibrator for ambiguous tasks. |
| title | Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.03154 |