SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929547284316160 |
|---|---|
| author | Ma, Zeyao Zhang, Bohan Zhang, Jing Yu, Jifan Zhang, Xiaokang Zhang, Xiaohan Luo, Sijia Wang, Xi Tang, Jie |
| author_facet | Ma, Zeyao Zhang, Bohan Zhang, Jing Yu, Jifan Zhang, Xiaokang Zhang, Xiaohan Luo, Sijia Wang, Xi Tang, Jie |
| contents | We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14991 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation Ma, Zeyao Zhang, Bohan Zhang, Jing Yu, Jifan Zhang, Xiaokang Zhang, Xiaohan Luo, Sijia Wang, Xi Tang, Jie Computation and Language Software Engineering We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty. |
| title | SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2406.14991 |