Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2511.12300 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908656422879232 |
|---|---|
| author | Sugiura, Naoya Yamada, Kosuke Ogawa, Yasuhiro Toyama, Katsuhiko Sasano, Ryohei |
| author_facet | Sugiura, Naoya Yamada, Kosuke Ogawa, Yasuhiro Toyama, Katsuhiko Sasano, Ryohei |
| contents | LLMs have achieved performance that surpasses humans in many NLP tasks. However, it remains unclear whether problems that are difficult for humans are also difficult for LLMs. This study investigates how the difficulty of quizzes in a buzzer setting differs between LLMs and humans. Specifically, we first collect Japanese quiz data including questions, answers, and correct response rate of humans, then prompted LLMs to answer the quizzes under several settings, and compare their correct answer rate to that of humans from two analytical perspectives. The experimental results showed that, compared to humans, LLMs struggle more with quizzes whose correct answers are not covered by Wikipedia entries, and also have difficulty with questions that require numerical answers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12300 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do LLMs and Humans Find the Same Questions Difficult? A Case Study on Japanese Quiz Answering Sugiura, Naoya Yamada, Kosuke Ogawa, Yasuhiro Toyama, Katsuhiko Sasano, Ryohei Computation and Language LLMs have achieved performance that surpasses humans in many NLP tasks. However, it remains unclear whether problems that are difficult for humans are also difficult for LLMs. This study investigates how the difficulty of quizzes in a buzzer setting differs between LLMs and humans. Specifically, we first collect Japanese quiz data including questions, answers, and correct response rate of humans, then prompted LLMs to answer the quizzes under several settings, and compare their correct answer rate to that of humans from two analytical perspectives. The experimental results showed that, compared to humans, LLMs struggle more with quizzes whose correct answers are not covered by Wikipedia entries, and also have difficulty with questions that require numerical answers. |
| title | Do LLMs and Humans Find the Same Questions Difficult? A Case Study on Japanese Quiz Answering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.12300 |