Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.15165 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911125869690880 |
|---|---|
| author | Wang, Yuanchun Yu, Jifan Yao, Zijun Zhang, Jing Xie, Yuyang Tu, Shangqing Fu, Yiyang Feng, Youhe Zhang, Jinkai Zhang, Jingyao Huang, Bowen Li, Yuanyao Yuan, Huihui Hou, Lei Li, Juanzi Tang, Jie |
| author_facet | Wang, Yuanchun Yu, Jifan Yao, Zijun Zhang, Jing Xie, Yuyang Tu, Shangqing Fu, Yiyang Feng, Youhe Zhang, Jinkai Zhang, Jingyao Huang, Bowen Li, Yuanyao Yuan, Huihui Hou, Lei Li, Juanzi Tang, Jie |
| contents | Applying large language models (LLMs) for academic API usage shows promise in reducing researchers' academic information seeking efforts. However, current LLM API-using methods struggle with complex API coupling commonly encountered in academic queries. To address this, we introduce SoAy, a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. The addition of the solution reduces the difficulty for the model to understand the complex relationships between APIs. Code improves the efficiency of reasoning.
To evaluate SoAy, we introduce SoAyBench, an evaluation benchmark accompanied by SoAyEval, built upon a cloned environment of APIs from AMiner. Experimental results demonstrate a 34.58-75.99\% performance improvement compared to state-of-the-art LLM API-based baselines. All datasets, codes, tuned models, and deployed online services are publicly accessible at https://github.com/RUCKBReasoning/SoAy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15165 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking Wang, Yuanchun Yu, Jifan Yao, Zijun Zhang, Jing Xie, Yuyang Tu, Shangqing Fu, Yiyang Feng, Youhe Zhang, Jinkai Zhang, Jingyao Huang, Bowen Li, Yuanyao Yuan, Huihui Hou, Lei Li, Juanzi Tang, Jie Computation and Language Artificial Intelligence Software Engineering Applying large language models (LLMs) for academic API usage shows promise in reducing researchers' academic information seeking efforts. However, current LLM API-using methods struggle with complex API coupling commonly encountered in academic queries. To address this, we introduce SoAy, a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. The addition of the solution reduces the difficulty for the model to understand the complex relationships between APIs. Code improves the efficiency of reasoning. To evaluate SoAy, we introduce SoAyBench, an evaluation benchmark accompanied by SoAyEval, built upon a cloned environment of APIs from AMiner. Experimental results demonstrate a 34.58-75.99\% performance improvement compared to state-of-the-art LLM API-based baselines. All datasets, codes, tuned models, and deployed online services are publicly accessible at https://github.com/RUCKBReasoning/SoAy. |
| title | SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking |
| topic | Computation and Language Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2405.15165 |