OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909227579080704 |
|---|---|
| author | Zhang, Fanjin Shi, Shijie Zhu, Yifan Chen, Bo Cen, Yukuo Yu, Jifan Chen, Yelin Wang, Lulu Zhao, Qingfei Cheng, Yuqing Han, Tianyi An, Yuwei Zhang, Dan Tam, Weng Lam Cao, Kun Pang, Yunhe Guan, Xinyu Yuan, Huihui Song, Jian Li, Xiaoyan Dong, Yuxiao Tang, Jie |
| author_facet | Zhang, Fanjin Shi, Shijie Zhu, Yifan Chen, Bo Cen, Yukuo Yu, Jifan Chen, Yelin Wang, Lulu Zhao, Qingfei Cheng, Yuqing Han, Tianyi An, Yuwei Zhang, Dan Tam, Weng Lam Cao, Kun Pang, Yunhe Guan, Xinyu Yuan, Huihui Song, Jian Li, Xiaoyan Dong, Yuxiao Tang, Jie |
| contents | With the rapid proliferation of scientific literature, versatile academic knowledge services increasingly rely on comprehensive academic graph mining. Despite the availability of public academic graphs, benchmarks, and datasets, these resources often fall short in multi-aspect and fine-grained annotations, are constrained to specific task types and domains, or lack underlying real academic graphs. In this paper, we present OAG-Bench, a comprehensive, multi-aspect, and fine-grained human-curated benchmark based on the Open Academic Graph (OAG). OAG-Bench covers 10 tasks, 20 datasets, 70+ baselines, and 120+ experimental results to date. We propose new data annotation strategies for certain tasks and offer a suite of data pre-processing codes, algorithm implementations, and standardized evaluation protocols to facilitate academic graph mining. Extensive experiments reveal that even advanced algorithms like large language models (LLMs) encounter difficulties in addressing key challenges in certain tasks, such as paper source tracing and scholar profiling. We also introduce the Open Academic Graph Challenge (OAG-Challenge) to encourage community input and sharing. We envisage that OAG-Bench can serve as a common ground for the community to evaluate and compare algorithms in academic graph mining, thereby accelerating algorithm development and advancement in this field. OAG-Bench is accessible at https://www.aminer.cn/data/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15810 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining Zhang, Fanjin Shi, Shijie Zhu, Yifan Chen, Bo Cen, Yukuo Yu, Jifan Chen, Yelin Wang, Lulu Zhao, Qingfei Cheng, Yuqing Han, Tianyi An, Yuwei Zhang, Dan Tam, Weng Lam Cao, Kun Pang, Yunhe Guan, Xinyu Yuan, Huihui Song, Jian Li, Xiaoyan Dong, Yuxiao Tang, Jie Digital Libraries Computation and Language Machine Learning With the rapid proliferation of scientific literature, versatile academic knowledge services increasingly rely on comprehensive academic graph mining. Despite the availability of public academic graphs, benchmarks, and datasets, these resources often fall short in multi-aspect and fine-grained annotations, are constrained to specific task types and domains, or lack underlying real academic graphs. In this paper, we present OAG-Bench, a comprehensive, multi-aspect, and fine-grained human-curated benchmark based on the Open Academic Graph (OAG). OAG-Bench covers 10 tasks, 20 datasets, 70+ baselines, and 120+ experimental results to date. We propose new data annotation strategies for certain tasks and offer a suite of data pre-processing codes, algorithm implementations, and standardized evaluation protocols to facilitate academic graph mining. Extensive experiments reveal that even advanced algorithms like large language models (LLMs) encounter difficulties in addressing key challenges in certain tasks, such as paper source tracing and scholar profiling. We also introduce the Open Academic Graph Challenge (OAG-Challenge) to encourage community input and sharing. We envisage that OAG-Bench can serve as a common ground for the community to evaluate and compare algorithms in academic graph mining, thereby accelerating algorithm development and advancement in this field. OAG-Bench is accessible at https://www.aminer.cn/data/. |
| title | OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining |
| topic | Digital Libraries Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.15810 |