TableBench: A Comprehensive and Complex Benchmark for Table Question Answering
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917959294779392 |
|---|---|
| author | Wu, Xianjie Yang, Jian Chai, Linzheng Zhang, Ge Liu, Jiaheng Du, Xinrun Liang, Di Shu, Daixin Cheng, Xianfu Sun, Tianzhen Niu, Guanglin Li, Tongliang Li, Zhoujun |
| author_facet | Wu, Xianjie Yang, Jian Chai, Linzheng Zhang, Ge Liu, Jiaheng Du, Xinrun Liang, Di Shu, Daixin Cheng, Xianfu Sun, Tianzhen Niu, Guanglin Li, Tongliang Li, Zhoujun |
| contents | Recent advancements in Large Language Models (LLMs) have markedly enhanced the interpretation and processing of tabular data, introducing previously unimaginable capabilities. Despite these achievements, LLMs still encounter significant challenges when applied in industrial scenarios, particularly due to the increased complexity of reasoning required with real-world tabular data, underscoring a notable disparity between academic benchmarks and practical applications. To address this discrepancy, we conduct a detailed investigation into the application of tabular data in industrial scenarios and propose a comprehensive and complex benchmark TableBench, including 18 fields within four major categories of table question answering (TableQA) capabilities. Furthermore, we introduce TableLLM, trained on our meticulously constructed training set TableInstruct, achieving comparable performance with GPT-3.5. Massive experiments conducted on TableBench indicate that both open-source and proprietary LLMs still have significant room for improvement to meet real-world demands, where the most advanced model, GPT-4, achieves only a modest score compared to humans. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09174 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TableBench: A Comprehensive and Complex Benchmark for Table Question Answering Wu, Xianjie Yang, Jian Chai, Linzheng Zhang, Ge Liu, Jiaheng Du, Xinrun Liang, Di Shu, Daixin Cheng, Xianfu Sun, Tianzhen Niu, Guanglin Li, Tongliang Li, Zhoujun Computation and Language Recent advancements in Large Language Models (LLMs) have markedly enhanced the interpretation and processing of tabular data, introducing previously unimaginable capabilities. Despite these achievements, LLMs still encounter significant challenges when applied in industrial scenarios, particularly due to the increased complexity of reasoning required with real-world tabular data, underscoring a notable disparity between academic benchmarks and practical applications. To address this discrepancy, we conduct a detailed investigation into the application of tabular data in industrial scenarios and propose a comprehensive and complex benchmark TableBench, including 18 fields within four major categories of table question answering (TableQA) capabilities. Furthermore, we introduce TableLLM, trained on our meticulously constructed training set TableInstruct, achieving comparable performance with GPT-3.5. Massive experiments conducted on TableBench indicate that both open-source and proprietary LLMs still have significant room for improvement to meet real-world demands, where the most advanced model, GPT-4, achieves only a modest score compared to humans. |
| title | TableBench: A Comprehensive and Complex Benchmark for Table Question Answering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.09174 |