Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.07853 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918436465016832 |
|---|---|
| author | Gu, Hao Wang, Hao Liu, Jiacheng Li, Lujun Zhu, Qiyuan Liu, Bei Xu, Binxing Wang, Lei Yang, Xintong Lin, Sida Han, Sirui Guo, Yike |
| author_facet | Gu, Hao Wang, Hao Liu, Jiacheng Li, Lujun Zhu, Qiyuan Liu, Bei Xu, Binxing Wang, Lei Yang, Xintong Lin, Sida Han, Sirui Guo, Yike |
| contents | Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07853 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch Gu, Hao Wang, Hao Liu, Jiacheng Li, Lujun Zhu, Qiyuan Liu, Bei Xu, Binxing Wang, Lei Yang, Xintong Lin, Sida Han, Sirui Guo, Yike Machine Learning Artificial Intelligence Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits. |
| title | QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.07853 |