Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908428403736576 |
|---|---|
| author | Hu, Nan Chen, Jiaoyan Wu, Yike Qi, Guilin Wang, Hongru Bi, Sheng Chen, Yongrui Wu, Tongtong Pan, Jeff Z. |
| author_facet | Hu, Nan Chen, Jiaoyan Wu, Yike Qi, Guilin Wang, Hongru Bi, Sheng Chen, Yongrui Wu, Tongtong Pan, Jeff Z. |
| contents | Attributed Question Answering (AQA) has attracted wide attention, but there are still several limitations in evaluating the attributions, including lacking fine-grained attribution categories, relying on manual annotations, and failing to compare attributions with only subtle differences. To bridge these gaps, we introduce Complex Attributed Question Answering (CAQA), a large-scale benchmark containing comprehensive attribution categories, automatically generated using Knowledge Graphs (KGs), and complex attribution scenarios. We have conducted extensive experiments to verify the effectiveness of CAQA, including the benchmarking of 25 automatic evaluators, their comparison with human evaluators, the testing of LLM evaluators fine-tuned by CAQA and so on. These experiments also lead to a series of important findings that can benefit the future research of AQA. All the codes and data are publicly accessible at https://github.com/HuuuNan/CAQA-Benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_14640 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs Hu, Nan Chen, Jiaoyan Wu, Yike Qi, Guilin Wang, Hongru Bi, Sheng Chen, Yongrui Wu, Tongtong Pan, Jeff Z. Computation and Language Attributed Question Answering (AQA) has attracted wide attention, but there are still several limitations in evaluating the attributions, including lacking fine-grained attribution categories, relying on manual annotations, and failing to compare attributions with only subtle differences. To bridge these gaps, we introduce Complex Attributed Question Answering (CAQA), a large-scale benchmark containing comprehensive attribution categories, automatically generated using Knowledge Graphs (KGs), and complex attribution scenarios. We have conducted extensive experiments to verify the effectiveness of CAQA, including the benchmarking of 25 automatic evaluators, their comparison with human evaluators, the testing of LLM evaluators fine-tuned by CAQA and so on. These experiments also lead to a series of important findings that can benefit the future research of AQA. All the codes and data are publicly accessible at https://github.com/HuuuNan/CAQA-Benchmark. |
| title | Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.14640 |