STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910504209874944 |
|---|---|
| author | Li, Wenbin Yao, Di Zhao, Ruibo Chen, Wenjie Xu, Zijie Luo, Chengxue Gong, Chang Jing, Quanliang Tan, Haining Bi, Jingping |
| author_facet | Li, Wenbin Yao, Di Zhao, Ruibo Chen, Wenjie Xu, Zijie Luo, Chengxue Gong, Chang Jing, Quanliang Tan, Haining Bi, Jingping |
| contents | The rapid evolution of large language models (LLMs) holds promise for reforming the methodology of spatio-temporal data mining. However, current works for evaluating the spatio-temporal understanding capability of LLMs are somewhat limited and biased. These works either fail to incorporate the latest language models or only focus on assessing the memorized spatio-temporal knowledge. To address this gap, this paper dissects LLMs' capability of spatio-temporal data into four distinct dimensions: knowledge comprehension, spatio-temporal reasoning, accurate computation, and downstream applications. We curate several natural language question-answer tasks for each category and build the benchmark dataset, namely STBench, containing 13 distinct tasks and over 60,000 QA pairs. Moreover, we have assessed the capabilities of 13 LLMs, such as GPT-4o, Gemma and Mistral. Experimental results reveal that existing LLMs show remarkable performance on knowledge comprehension and spatio-temporal reasoning tasks, with potential for further enhancement on other tasks through in-context learning, chain-of-though prompting, and fine-tuning. The code and datasets of STBench are released on https://github.com/LwbXc/STBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19065 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis Li, Wenbin Yao, Di Zhao, Ruibo Chen, Wenjie Xu, Zijie Luo, Chengxue Gong, Chang Jing, Quanliang Tan, Haining Bi, Jingping Computation and Language The rapid evolution of large language models (LLMs) holds promise for reforming the methodology of spatio-temporal data mining. However, current works for evaluating the spatio-temporal understanding capability of LLMs are somewhat limited and biased. These works either fail to incorporate the latest language models or only focus on assessing the memorized spatio-temporal knowledge. To address this gap, this paper dissects LLMs' capability of spatio-temporal data into four distinct dimensions: knowledge comprehension, spatio-temporal reasoning, accurate computation, and downstream applications. We curate several natural language question-answer tasks for each category and build the benchmark dataset, namely STBench, containing 13 distinct tasks and over 60,000 QA pairs. Moreover, we have assessed the capabilities of 13 LLMs, such as GPT-4o, Gemma and Mistral. Experimental results reveal that existing LLMs show remarkable performance on knowledge comprehension and spatio-temporal reasoning tasks, with potential for further enhancement on other tasks through in-context learning, chain-of-though prompting, and fine-tuning. The code and datasets of STBench are released on https://github.com/LwbXc/STBench. |
| title | STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.19065 |