Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.14783 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908561917870080 |
|---|---|
| author | Li, Derek Zhou, Jiaming Brunswic, Leo Maxime Ghaddar, Abbas Sun, Qianyi Ma, Liheng Luo, Yu Li, Dong Coates, Mark Hao, Jianye Zhang, Yingxue |
| author_facet | Li, Derek Zhou, Jiaming Brunswic, Leo Maxime Ghaddar, Abbas Sun, Qianyi Ma, Liheng Luo, Yu Li, Dong Coates, Mark Hao, Jianye Zhang, Yingxue |
| contents | The pursuit of general-purpose artificial intelligence depends on large language models (LLMs) that can handle both structured reasoning and open-ended generation. We present Omni-Thinker, a unified reinforcement learning (RL) framework that scales LLMs across diverse tasks by combining hybrid rewards with backward-transfer-guided scheduling. Hybrid rewards integrate rule-based verifiable signals with preference-based evaluations from an LLM-as-a-Judge, enabling learning in both deterministic and subjective domains. Our scheduler orders tasks according to accuracy backward transfer (BWT), reducing forgetting and improving multi-task performance. Experiments across four domains show gains of 6.2% over joint training and 12.4% over model merging. Moreover, we demonstrate that simple assumptions on accuracy transfer yield accurate predictions of curriculum outcomes, with entropy dynamics explaining deviations due to generative tasks. These findings underscore the importance of BWT-aware scheduling and hybrid supervision for scaling RL-based post-training toward general-purpose LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14783 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling Li, Derek Zhou, Jiaming Brunswic, Leo Maxime Ghaddar, Abbas Sun, Qianyi Ma, Liheng Luo, Yu Li, Dong Coates, Mark Hao, Jianye Zhang, Yingxue Machine Learning Artificial Intelligence The pursuit of general-purpose artificial intelligence depends on large language models (LLMs) that can handle both structured reasoning and open-ended generation. We present Omni-Thinker, a unified reinforcement learning (RL) framework that scales LLMs across diverse tasks by combining hybrid rewards with backward-transfer-guided scheduling. Hybrid rewards integrate rule-based verifiable signals with preference-based evaluations from an LLM-as-a-Judge, enabling learning in both deterministic and subjective domains. Our scheduler orders tasks according to accuracy backward transfer (BWT), reducing forgetting and improving multi-task performance. Experiments across four domains show gains of 6.2% over joint training and 12.4% over model merging. Moreover, we demonstrate that simple assumptions on accuracy transfer yield accurate predictions of curriculum outcomes, with entropy dynamics explaining deviations due to generative tasks. These findings underscore the importance of BWT-aware scheduling and hybrid supervision for scaling RL-based post-training toward general-purpose LLMs. |
| title | Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.14783 |