When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910081760624640 |
|---|---|
| author | Yasir, Tahreem Tithi, Sutapa Dey Tabarsi, Benyamin Droujkov, Dmitri Rajapaksha, Sam Gilson Yasitha Tian, Xiaoyi Ramesh, Arun DongKuan Xu Barnes, Tiffany |
| author_facet | Yasir, Tahreem Tithi, Sutapa Dey Tabarsi, Benyamin Droujkov, Dmitri Rajapaksha, Sam Gilson Yasitha Tian, Xiaoyi Ramesh, Arun DongKuan Xu Barnes, Tiffany |
| contents | Large language models (LLMs) are increasingly used for automated tutoring, but their reliability in structured symbolic domains remains unclear. We study step-level feedback for propositional logic proofs, which require precise symbolic reasoning aligned with a learner's current proof state. We introduce a knowledge-graph-grounded benchmark of 516 unique proof states with step-level annotations and difficulty metrics. Unlike prior tutoring evaluations that rely on model self-assessment or binary correctness, our framework enables fine-grained analysis of feedback quality against verified solution paths. We evaluate three role-specialized pipelines with varying solution access: Tutor (partial solution access), Teacher (full derivation access), and Judge (verification of Tutor feedback). Our results reveal a striking asymmetry: verification improves outcomes when upstream feedback is error-prone (<70% accuracy), but degrades performance by 4-6 percentage points through over-specification when feedback is already reliable (>85%). Critically, we identify a shared complexity ceiling; no model or pipeline reliably succeeds on proof states exceeding complexity 4-5. These findings challenge the assumption that adding verifiers or richer context universally improves tutoring, motivating adaptive, difficulty-aware architectures that route problems by estimated complexity and upstream reliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_27076 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring Yasir, Tahreem Tithi, Sutapa Dey Tabarsi, Benyamin Droujkov, Dmitri Rajapaksha, Sam Gilson Yasitha Tian, Xiaoyi Ramesh, Arun DongKuan Xu Barnes, Tiffany Artificial Intelligence Large language models (LLMs) are increasingly used for automated tutoring, but their reliability in structured symbolic domains remains unclear. We study step-level feedback for propositional logic proofs, which require precise symbolic reasoning aligned with a learner's current proof state. We introduce a knowledge-graph-grounded benchmark of 516 unique proof states with step-level annotations and difficulty metrics. Unlike prior tutoring evaluations that rely on model self-assessment or binary correctness, our framework enables fine-grained analysis of feedback quality against verified solution paths. We evaluate three role-specialized pipelines with varying solution access: Tutor (partial solution access), Teacher (full derivation access), and Judge (verification of Tutor feedback). Our results reveal a striking asymmetry: verification improves outcomes when upstream feedback is error-prone (<70% accuracy), but degrades performance by 4-6 percentage points through over-specification when feedback is already reliable (>85%). Critically, we identify a shared complexity ceiling; no model or pipeline reliably succeeds on proof states exceeding complexity 4-5. These findings challenge the assumption that adding verifiers or richer context universally improves tutoring, motivating adaptive, difficulty-aware architectures that route problems by estimated complexity and upstream reliability. |
| title | When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.27076 |