Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915772447588352 |
|---|---|
| author | Di, Zixiang Han, Jinyi Zhang, Shuo Liao, Ying Li, Zhi Ji, Xiaofeng Wang, Yongqi Yang, Zheming Gao, Ming Li, Bingdong Wang, Jie |
| author_facet | Di, Zixiang Han, Jinyi Zhang, Shuo Liao, Ying Li, Zhi Ji, Xiaofeng Wang, Yongqi Yang, Zheming Gao, Ming Li, Bingdong Wang, Jie |
| contents | Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03516 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning Di, Zixiang Han, Jinyi Zhang, Shuo Liao, Ying Li, Zhi Ji, Xiaofeng Wang, Yongqi Yang, Zheming Gao, Ming Li, Bingdong Wang, Jie Machine Learning Artificial Intelligence Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models. |
| title | Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.03516 |