Argument-Based Comparative Question Answering Evaluation Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910837065646080 |
|---|---|
| author | Nikishina, Irina Anwar, Saba Dolgov, Nikolay Manina, Maria Ignatenko, Daria Moskvoretskii, Viktor Shelmanov, Artem Baldwin, Tim Biemann, Chris |
| author_facet | Nikishina, Irina Anwar, Saba Dolgov, Nikolay Manina, Maria Ignatenko, Daria Moskvoretskii, Viktor Shelmanov, Artem Baldwin, Tim Biemann, Chris |
| contents | In this paper, we aim to solve the problems standing in the way of automatic comparative question answering. To this end, we propose an evaluation framework to assess the quality of comparative question answering summaries. We formulate 15 criteria for assessing comparative answers created using manual annotation and annotation from 6 large language models and two comparative question asnwering datasets. We perform our tests using several LLMs and manual annotation under different settings and demonstrate the constituency of both evaluations. Our results demonstrate that the Llama-3 70B Instruct model demonstrates the best results for summary evaluation, while GPT-4 is the best for answering comparative questions. All used data, code, and evaluation results are publicly available\footnote{\url{https://anonymous.4open.science/r/cqa-evaluation-benchmark-4561/README.md}}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14476 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Argument-Based Comparative Question Answering Evaluation Benchmark Nikishina, Irina Anwar, Saba Dolgov, Nikolay Manina, Maria Ignatenko, Daria Moskvoretskii, Viktor Shelmanov, Artem Baldwin, Tim Biemann, Chris Computation and Language In this paper, we aim to solve the problems standing in the way of automatic comparative question answering. To this end, we propose an evaluation framework to assess the quality of comparative question answering summaries. We formulate 15 criteria for assessing comparative answers created using manual annotation and annotation from 6 large language models and two comparative question asnwering datasets. We perform our tests using several LLMs and manual annotation under different settings and demonstrate the constituency of both evaluations. Our results demonstrate that the Llama-3 70B Instruct model demonstrates the best results for summary evaluation, while GPT-4 is the best for answering comparative questions. All used data, code, and evaluation results are publicly available\footnote{\url{https://anonymous.4open.science/r/cqa-evaluation-benchmark-4561/README.md}}. |
| title | Argument-Based Comparative Question Answering Evaluation Benchmark |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.14476 |