OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916436909228032 |
|---|---|
| author | Wang, Jun Fang, Meng Wan, Ziyu Wen, Muning Zhu, Jiachen Liu, Anjie Gong, Ziqin Song, Yan Chen, Lei Ni, Lionel M. Yang, Linyi Wen, Ying Zhang, Weinan |
| author_facet | Wang, Jun Fang, Meng Wan, Ziyu Wen, Muning Zhu, Jiachen Liu, Anjie Gong, Ziqin Song, Yan Chen, Lei Ni, Lionel M. Yang, Linyi Wen, Ying Zhang, Weinan |
| contents | In this technical report, we introduce OpenR, an open-source framework designed to integrate key components for enhancing the reasoning capabilities of large language models (LLMs). OpenR unifies data acquisition, reinforcement learning training (both online and offline), and non-autoregressive decoding into a cohesive software platform. Our goal is to establish an open-source platform and community to accelerate the development of LLM reasoning. Inspired by the success of OpenAI's o1 model, which demonstrated improved reasoning abilities through step-by-step reasoning and reinforcement learning, OpenR integrates test-time compute, reinforcement learning, and process supervision to improve reasoning in LLMs. Our work is the first to provide an open-source framework that explores the core techniques of OpenAI's o1 model with reinforcement learning, achieving advanced reasoning capabilities beyond traditional autoregressive methods. We demonstrate the efficacy of OpenR by evaluating it on the MATH dataset, utilising publicly available data and search methods. Our initial experiments confirm substantial gains, with relative improvements in reasoning and performance driven by test-time computation and reinforcement learning through process reward models. The OpenR framework, including code, models, and datasets, is accessible at https://openreasoner.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models Wang, Jun Fang, Meng Wan, Ziyu Wen, Muning Zhu, Jiachen Liu, Anjie Gong, Ziqin Song, Yan Chen, Lei Ni, Lionel M. Yang, Linyi Wen, Ying Zhang, Weinan Artificial Intelligence Computation and Language In this technical report, we introduce OpenR, an open-source framework designed to integrate key components for enhancing the reasoning capabilities of large language models (LLMs). OpenR unifies data acquisition, reinforcement learning training (both online and offline), and non-autoregressive decoding into a cohesive software platform. Our goal is to establish an open-source platform and community to accelerate the development of LLM reasoning. Inspired by the success of OpenAI's o1 model, which demonstrated improved reasoning abilities through step-by-step reasoning and reinforcement learning, OpenR integrates test-time compute, reinforcement learning, and process supervision to improve reasoning in LLMs. Our work is the first to provide an open-source framework that explores the core techniques of OpenAI's o1 model with reinforcement learning, achieving advanced reasoning capabilities beyond traditional autoregressive methods. We demonstrate the efficacy of OpenR by evaluating it on the MATH dataset, utilising publicly available data and search methods. Our initial experiments confirm substantial gains, with relative improvements in reasoning and performance driven by test-time computation and reinforcement learning through process reward models. The OpenR framework, including code, models, and datasets, is accessible at https://openreasoner.github.io. |
| title | OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.09671 |