Efficient RL Training for LLMs with Experience Replay
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915929591382016 |
|---|---|
| author | Arnal, Charles Cabannes, Vivien Cohen, Taco Kempe, Julia Munos, Remi |
| author_facet | Arnal, Charles Cabannes, Vivien Cohen, Taco Kempe, Julia Munos, Remi |
| contents | While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08706 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Efficient RL Training for LLMs with Experience Replay Arnal, Charles Cabannes, Vivien Cohen, Taco Kempe, Julia Munos, Remi Machine Learning While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy. |
| title | Efficient RL Training for LLMs with Experience Replay |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.08706 |