AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914568807120896 |
|---|---|
| author | Zheng, Haizhong Di, Yizhuo Wang, Jiahui Jin, Shuowei Liu, Xueshen Wu, Yongji Mao, Z. Morley Stoica, Ion Zhao, Jiawei Chen, Beidi |
| author_facet | Zheng, Haizhong Di, Yizhuo Wang, Jiahui Jin, Shuowei Liu, Xueshen Wu, Yongji Mao, Z. Morley Stoica, Ion Zhao, Jiawei Chen, Beidi |
| contents | Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, while efficiently using elastic, heterogeneous, and cross-region compute resources. Existing LLM RL systems support some of these capabilities, but each new extension often requires dedicated system engineering. This burden arises from trainer-centered control architectures and the lack of principled abstractions for RL system components. To address these limitations, we propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, enabling the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources. We evaluate AstraFlow across math, code, search, and AgentBench workloads, showing that the same system supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without system-level code changes. In multi-policy collaborative training, AstraFlow achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15565 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs Zheng, Haizhong Di, Yizhuo Wang, Jiahui Jin, Shuowei Liu, Xueshen Wu, Yongji Mao, Z. Morley Stoica, Ion Zhao, Jiawei Chen, Beidi Machine Learning Artificial Intelligence Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, while efficiently using elastic, heterogeneous, and cross-region compute resources. Existing LLM RL systems support some of these capabilities, but each new extension often requires dedicated system engineering. This burden arises from trainer-centered control architectures and the lack of principled abstractions for RL system components. To address these limitations, we propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, enabling the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources. We evaluate AstraFlow across math, code, search, and AgentBench workloads, showing that the same system supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without system-level code changes. In multi-policy collaborative training, AstraFlow achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7x. |
| title | AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.15565 |