Multi-Hop Question Answering: When Can Humans Help, and Where do They Struggle?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908577396948992 |
|---|---|
| author | Su, Jinyan Cardie, Claire Healey, Jennifer |
| author_facet | Su, Jinyan Cardie, Claire Healey, Jennifer |
| contents | Multi-hop question answering is a challenging task for both large language models (LLMs) and humans, as it requires recognizing when multi-hop reasoning is needed, followed by reading comprehension, logical reasoning, and knowledge integration. To better understand how humans might collaborate effectively with AI, we evaluate the performance of crowd workers on these individual reasoning subtasks. We find that while humans excel at knowledge integration (97\% accuracy), they often fail to recognize when a question requires multi-hop reasoning (67\% accuracy). Participants perform reasonably well on both single-hop and multi-hop QA (84\% and 80\% accuracy, respectively), but frequently make semantic mistakes--for example, answering "when" an event happened when the question asked "where." These findings highlight the importance of designing AI systems that complement human strengths while compensating for common weaknesses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04493 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Hop Question Answering: When Can Humans Help, and Where do They Struggle? Su, Jinyan Cardie, Claire Healey, Jennifer Human-Computer Interaction Multi-hop question answering is a challenging task for both large language models (LLMs) and humans, as it requires recognizing when multi-hop reasoning is needed, followed by reading comprehension, logical reasoning, and knowledge integration. To better understand how humans might collaborate effectively with AI, we evaluate the performance of crowd workers on these individual reasoning subtasks. We find that while humans excel at knowledge integration (97\% accuracy), they often fail to recognize when a question requires multi-hop reasoning (67\% accuracy). Participants perform reasonably well on both single-hop and multi-hop QA (84\% and 80\% accuracy, respectively), but frequently make semantic mistakes--for example, answering "when" an event happened when the question asked "where." These findings highlight the importance of designing AI systems that complement human strengths while compensating for common weaknesses. |
| title | Multi-Hop Question Answering: When Can Humans Help, and Where do They Struggle? |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2510.04493 |