Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2510.20279 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909866978705408 |
|---|---|
| author | Wang, Penghao Zhou, Yuhao Wu, Mengxuan Qin, Ziheng Zhu, Bangyuan Huang, Shengbin Zhao, Xuanlei Zhang, Panpan Peng, Xiaojiang Shang, Yuzhang Yang, Jianfei Zhu, Zheng Chen, Tianlong Wang, Zhangyang Wang, Kai |
| author_facet | Wang, Penghao Zhou, Yuhao Wu, Mengxuan Qin, Ziheng Zhu, Bangyuan Huang, Shengbin Zhao, Xuanlei Zhang, Panpan Peng, Xiaojiang Shang, Yuzhang Yang, Jianfei Zhu, Zheng Chen, Tianlong Wang, Zhangyang Wang, Kai |
| contents | As large language models (LLMs) advance, the ultimate vision for their role in science is emerging: we could build an AI collaborator to effectively assist human beings throughout the entire scientific research process. We refer to this envisioned system as ResearchGPT. Given that scientific research progresses through multiple interdependent phases, achieving this vision requires rigorous benchmarks that evaluate the end-to-end workflow rather than isolated sub-tasks. To this end, we contribute CS-54k, a high-quality corpus of scientific Q&A pairs in computer science, built from 14k CC-licensed papers. It is constructed through a scalable, paper-grounded pipeline that combines retrieval-augmented generation (RAG) with multi-stage quality control to ensure factual grounding. From this unified corpus, we derive two complementary subsets: CS-4k, a carefully curated benchmark for evaluating AI's ability to assist scientific research, and CS-50k, a large-scale training dataset. Extensive experiments demonstrate that CS-4k stratifies state-of-the-art LLMs into distinct capability tiers. Open models trained on CS-50k with supervised training and reinforcement learning demonstrate substantial improvements. Even 7B-scale models, when properly trained, outperform many larger proprietary systems, such as GPT-4.1, GPT-4o, and Gemini 2.5 Pro. This indicates that making AI models better research assistants relies more on domain-aligned training with high-quality data than on pretraining scale or general benchmark performance. We release CS-4k and CS-50k in the hope of fostering AI systems as reliable collaborators in CS research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20279 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows Wang, Penghao Zhou, Yuhao Wu, Mengxuan Qin, Ziheng Zhu, Bangyuan Huang, Shengbin Zhao, Xuanlei Zhang, Panpan Peng, Xiaojiang Shang, Yuzhang Yang, Jianfei Zhu, Zheng Chen, Tianlong Wang, Zhangyang Wang, Kai Machine Learning As large language models (LLMs) advance, the ultimate vision for their role in science is emerging: we could build an AI collaborator to effectively assist human beings throughout the entire scientific research process. We refer to this envisioned system as ResearchGPT. Given that scientific research progresses through multiple interdependent phases, achieving this vision requires rigorous benchmarks that evaluate the end-to-end workflow rather than isolated sub-tasks. To this end, we contribute CS-54k, a high-quality corpus of scientific Q&A pairs in computer science, built from 14k CC-licensed papers. It is constructed through a scalable, paper-grounded pipeline that combines retrieval-augmented generation (RAG) with multi-stage quality control to ensure factual grounding. From this unified corpus, we derive two complementary subsets: CS-4k, a carefully curated benchmark for evaluating AI's ability to assist scientific research, and CS-50k, a large-scale training dataset. Extensive experiments demonstrate that CS-4k stratifies state-of-the-art LLMs into distinct capability tiers. Open models trained on CS-50k with supervised training and reinforcement learning demonstrate substantial improvements. Even 7B-scale models, when properly trained, outperform many larger proprietary systems, such as GPT-4.1, GPT-4o, and Gemini 2.5 Pro. This indicates that making AI models better research assistants relies more on domain-aligned training with high-quality data than on pretraining scale or general benchmark performance. We release CS-4k and CS-50k in the hope of fostering AI systems as reliable collaborators in CS research. |
| title | ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.20279 |