Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.28183 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910281006841856 |
|---|---|
| author | Nagl, Sebastian Mayrhofer, Ann-Kristin Heidebach, Martin Koçak, Aleyna Zettelmeier, Anne Breu, Elly Greiner, Angelina Milijas, Sofija Grabmair, Matthias |
| author_facet | Nagl, Sebastian Mayrhofer, Ann-Kristin Heidebach, Martin Koçak, Aleyna Zettelmeier, Anne Breu, Elly Greiner, Angelina Milijas, Sofija Grabmair, Matthias |
| contents | We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER dataset consists of three components: 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. We evaluate 12 contemporary LLM systems -- closed flagship, efficiency-oriented, and open-weight -- across automatic and judge-based metrics. On a controlled validation subset of timed human-written solutions under both unaided and human--AI co-creation conditions, we contextualise model performance against these human baselines. We introduce a rubric-aligned LLM-as-a-Judge framework cross-validated against a multi-rater human-grading protocol (three blind reviews plus one author-informed creator review per solution). Our results show that replacing a blind human reviewer with the LLM judge degrades agreement with the full human pool no more than removing that reviewer altogether (Calderon r=0.96 vs.~r=0.96, matched n=30), that closed-flagship systems lead the leaderboard across all corpora, and that human--AI co-creation substantially outperforms unaided human work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28183 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law Nagl, Sebastian Mayrhofer, Ann-Kristin Heidebach, Martin Koçak, Aleyna Zettelmeier, Anne Breu, Elly Greiner, Angelina Milijas, Sofija Grabmair, Matthias Computation and Language Artificial Intelligence We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER dataset consists of three components: 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. We evaluate 12 contemporary LLM systems -- closed flagship, efficiency-oriented, and open-weight -- across automatic and judge-based metrics. On a controlled validation subset of timed human-written solutions under both unaided and human--AI co-creation conditions, we contextualise model performance against these human baselines. We introduce a rubric-aligned LLM-as-a-Judge framework cross-validated against a multi-rater human-grading protocol (three blind reviews plus one author-informed creator review per solution). Our results show that replacing a blind human reviewer with the LLM judge degrades agreement with the full human pool no more than removing that reviewer altogether (Calderon r=0.96 vs.~r=0.96, matched n=30), that closed-flagship systems lead the leaderboard across all corpora, and that human--AI co-creation substantially outperforms unaided human work. |
| title | BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.28183 |