Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.06631 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915779914498048 |
|---|---|
| author | Brant, Thiago Kühn, Julien Pang, Jun |
| author_facet | Brant, Thiago Kühn, Julien Pang, Jun |
| contents | As Large Language Models (LLMs) are increasingly deployed to generate educational content, a critical safety question arises: can these models reliably estimate the difficulty of the questions they produce? Using Brazil's high-stakes ENEM exam as a testbed, we benchmark ten proprietary and open-weight LLMs against official Item Response Theory (IRT) parameters for 1,031 questions. We evaluate performance along three axes: absolute calibration, rank fidelity, and context sensitivity across learner backgrounds. Our results reveal a significant trade-off: while the best models achieve moderate rank correlation, they systematically underestimate difficulty and degrade significantly on multimodal items. Crucially, we find that models exhibit limited and inconsistent plasticity when prompted with student demographic cues, suggesting they are not yet ready for context-adaptive personalization. We conclude that LLMs function best as calibrated screeners rather than authoritative oracles, supporting an "evaluation-before-generation" pipeline for responsible assessment design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06631 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Estimating Exam Item Difficulty with LLMs: A Benchmark on Brazil's ENEM Corpus Brant, Thiago Kühn, Julien Pang, Jun Computers and Society 68T50 I.2.7; K.3.1; J.4 As Large Language Models (LLMs) are increasingly deployed to generate educational content, a critical safety question arises: can these models reliably estimate the difficulty of the questions they produce? Using Brazil's high-stakes ENEM exam as a testbed, we benchmark ten proprietary and open-weight LLMs against official Item Response Theory (IRT) parameters for 1,031 questions. We evaluate performance along three axes: absolute calibration, rank fidelity, and context sensitivity across learner backgrounds. Our results reveal a significant trade-off: while the best models achieve moderate rank correlation, they systematically underestimate difficulty and degrade significantly on multimodal items. Crucially, we find that models exhibit limited and inconsistent plasticity when prompted with student demographic cues, suggesting they are not yet ready for context-adaptive personalization. We conclude that LLMs function best as calibrated screeners rather than authoritative oracles, supporting an "evaluation-before-generation" pipeline for responsible assessment design. |
| title | Estimating Exam Item Difficulty with LLMs: A Benchmark on Brazil's ENEM Corpus |
| topic | Computers and Society 68T50 I.2.7; K.3.1; J.4 |
| url | https://arxiv.org/abs/2602.06631 |