Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929707212079104 |
|---|---|
| author | Zhuang, Yuchen Yang, Jingfeng Jiang, Haoming Liu, Xin Cheng, Kewei Lokegaonkar, Sanket Gao, Yifan Ping, Qing Liu, Tianyi Huang, Binxuan Li, Zheng Wang, Zhengyang Chen, Pei Wang, Ruijie Zhang, Rongzhi Zalmout, Nasser Nigam, Priyanka Yin, Bing Zhang, Chao |
| author_facet | Zhuang, Yuchen Yang, Jingfeng Jiang, Haoming Liu, Xin Cheng, Kewei Lokegaonkar, Sanket Gao, Yifan Ping, Qing Liu, Tianyi Huang, Binxuan Li, Zheng Wang, Zhengyang Chen, Pei Wang, Ruijie Zhang, Rongzhi Zalmout, Nasser Nigam, Priyanka Yin, Bing Zhang, Chao |
| contents | Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous agents typically rely on complex prompting or extensive fine-tuning, which often fails to introduce new capabilities while preserving strong generalizability. We introduce Hephaestus-Forge, the first large-scale pre-training corpus designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adapting to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories to strengthen intrinsic reasoning. To explore effective training protocols, we investigate scaling laws to identify the optimal recipe in data mixing ratios. By continual pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks, demonstrating the effectiveness of our pre-training corpus in enhancing fundamental agentic capabilities and generalization of LLMs to new tasks or environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06589 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training Zhuang, Yuchen Yang, Jingfeng Jiang, Haoming Liu, Xin Cheng, Kewei Lokegaonkar, Sanket Gao, Yifan Ping, Qing Liu, Tianyi Huang, Binxuan Li, Zheng Wang, Zhengyang Chen, Pei Wang, Ruijie Zhang, Rongzhi Zalmout, Nasser Nigam, Priyanka Yin, Bing Zhang, Chao Computation and Language Artificial Intelligence Machine Learning Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous agents typically rely on complex prompting or extensive fine-tuning, which often fails to introduce new capabilities while preserving strong generalizability. We introduce Hephaestus-Forge, the first large-scale pre-training corpus designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adapting to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories to strengthen intrinsic reasoning. To explore effective training protocols, we investigate scaling laws to identify the optimal recipe in data mixing ratios. By continual pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks, demonstrating the effectiveness of our pre-training corpus in enhancing fundamental agentic capabilities and generalization of LLMs to new tasks or environments. |
| title | Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.06589 |