Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914569897639936 |
|---|---|
| author | Yasir, Tahreem Li, Wenbo Gilson, Sam Tithi, Sutapa Dey Tian, Xiaoyi Barnes, Tiffany |
| author_facet | Yasir, Tahreem Li, Wenbo Gilson, Sam Tithi, Sutapa Dey Tian, Xiaoyi Barnes, Tiffany |
| contents | Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution--feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16207 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most Yasir, Tahreem Li, Wenbo Gilson, Sam Tithi, Sutapa Dey Tian, Xiaoyi Barnes, Tiffany Artificial Intelligence Computation and Language Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution--feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue. |
| title | Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.16207 |