NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911269453299712 |
|---|---|
| author | Zotos, Leonidas de Jong, Ivo Pascal Valdenegro-Toro, Matias Sburlea, Andreea Ioana Nissim, Malvina van Rijn, Hedderik |
| author_facet | Zotos, Leonidas de Jong, Ivo Pascal Valdenegro-Toro, Matias Sburlea, Andreea Ioana Nissim, Malvina van Rijn, Hedderik |
| contents | Estimating the difficulty of exam questions is essential for developing good exams, but professors are not always good at this task. We compare various Large Language Model-based methods with three professors in their ability to estimate what percentage of students will give correct answers on True/False exam questions in the areas of Neural Networks and Machine Learning. Our results show that the professors have limited ability to distinguish between easy and difficult questions and that they are outperformed by directly asking Gemini 2.5 to solve this task. Yet, we obtained even better results using uncertainties of the LLMs solving the questions in a supervised learning setting, using only 42 training samples. We conclude that supervised learning using LLM uncertainty can help professors better estimate the difficulty of exam questions, improving the quality of assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03294 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty Zotos, Leonidas de Jong, Ivo Pascal Valdenegro-Toro, Matias Sburlea, Andreea Ioana Nissim, Malvina van Rijn, Hedderik Computation and Language Artificial Intelligence Estimating the difficulty of exam questions is essential for developing good exams, but professors are not always good at this task. We compare various Large Language Model-based methods with three professors in their ability to estimate what percentage of students will give correct answers on True/False exam questions in the areas of Neural Networks and Machine Learning. Our results show that the professors have limited ability to distinguish between easy and difficult questions and that they are outperformed by directly asking Gemini 2.5 to solve this task. Yet, we obtained even better results using uncertainties of the LLMs solving the questions in a supervised learning setting, using only 42 training samples. We conclude that supervised learning using LLM uncertainty can help professors better estimate the difficulty of exam questions, improving the quality of assessment. |
| title | NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.03294 |