Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910144443449344 |
|---|---|
| author | Lei, Xuanyu Li, Chenliang Wu, Yuning Liu, Kaiming Shen, Weizhou Li, Peng Yan, Ming Huang, Fei Zhang, Ya-Qin Liu, Yang |
| author_facet | Lei, Xuanyu Li, Chenliang Wu, Yuning Liu, Kaiming Shen, Weizhou Li, Peng Yan, Ming Huang, Fei Zhang, Ya-Qin Liu, Yang |
| contents | Recent advances in Large Language Models(LLMs) have enabled strong performance in long-form writing, but current training paradigms remain limited: Supervised Fine-Tuning (SFT) remains constrained by data saturation and performance ceilings, while Reinforcement Learning with Verifiable Reward (RLVR), though successful in verifiable domains like math and code, cannot be directly migrated to open-ended long-form writing due to a lack of ground-truths. To further advance long-form writing, we present Writing-RL: an Adaptive Curriculum Reinforcement Learning framework to advance long-form writing capabilities beyond SFT. The framework consists of three key components: Margin-aware Data Selection strategy that prioritizes samples with high learning potential, Pairwise Comparison Reward mechanism that provides discriminative learning signals in the absence of verifiable rewards, and Dynamic Reference Scheduling approach, which plays a critical role by adaptively adjusting task difficulty based on evolving model performance. Experiments on 7B-scale writer models show that Writing-RL effectively improves long-form writing performance over strong SFT baselines. Furthermore, we observe that models trained with long-output RL generalize surprisingly well to long-input reasoning tasks, potentially offering a promising perspective for rethinking long-context training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05760 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning Lei, Xuanyu Li, Chenliang Wu, Yuning Liu, Kaiming Shen, Weizhou Li, Peng Yan, Ming Huang, Fei Zhang, Ya-Qin Liu, Yang Computation and Language Recent advances in Large Language Models(LLMs) have enabled strong performance in long-form writing, but current training paradigms remain limited: Supervised Fine-Tuning (SFT) remains constrained by data saturation and performance ceilings, while Reinforcement Learning with Verifiable Reward (RLVR), though successful in verifiable domains like math and code, cannot be directly migrated to open-ended long-form writing due to a lack of ground-truths. To further advance long-form writing, we present Writing-RL: an Adaptive Curriculum Reinforcement Learning framework to advance long-form writing capabilities beyond SFT. The framework consists of three key components: Margin-aware Data Selection strategy that prioritizes samples with high learning potential, Pairwise Comparison Reward mechanism that provides discriminative learning signals in the absence of verifiable rewards, and Dynamic Reference Scheduling approach, which plays a critical role by adaptively adjusting task difficulty based on evolving model performance. Experiments on 7B-scale writer models show that Writing-RL effectively improves long-form writing performance over strong SFT baselines. Furthermore, we observe that models trained with long-output RL generalize surprisingly well to long-input reasoning tasks, potentially offering a promising perspective for rethinking long-context training. |
| title | Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.05760 |