HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911739774238720 |
|---|---|
| author | Zheng, Minghui Chen, Hongxu Ren, Huimin Xin, Hongsheng Qu, Xiaoyang Wang, Ze Yang, Shuling Peng, Ziyu Zhang, Kaike Zhou, Pan Zhan, Kun |
| author_facet | Zheng, Minghui Chen, Hongxu Ren, Huimin Xin, Hongsheng Qu, Xiaoyang Wang, Ze Yang, Shuling Peng, Ziyu Zhang, Kaike Zhou, Pan Zhan, Kun |
| contents | Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01934 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression Zheng, Minghui Chen, Hongxu Ren, Huimin Xin, Hongsheng Qu, Xiaoyang Wang, Ze Yang, Shuling Peng, Ziyu Zhang, Kaike Zhou, Pan Zhan, Kun Machine Learning Computation and Language Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines. |
| title | HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2606.01934 |