Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914413153353728 |
|---|---|
| author | Cao, Hongyu Liu, Kunpeng Wang, Dongjie Fu, Yanjie |
| author_facet | Cao, Hongyu Liu, Kunpeng Wang, Dongjie Fu, Yanjie |
| contents | Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_20899 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach Cao, Hongyu Liu, Kunpeng Wang, Dongjie Fu, Yanjie Computation and Language Artificial Intelligence Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning. |
| title | Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.20899 |