LEXam: Benchmarking Legal Reasoning on 340 Law Exams
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911561007759360 |
|---|---|
| author | Fan, Yu Ni, Jingwei Merane, Jakob Tian, Yang Hermstrüwer, Yoan Huang, Yinya Akhtar, Mubashara Salimbeni, Etienne Geering, Florian Dreyer, Oliver Brunner, Daniel Leippold, Markus Sachan, Mrinmaya Stremitzer, Alexander Engel, Christoph Ash, Elliott Niklaus, Joel |
| author_facet | Fan, Yu Ni, Jingwei Merane, Jakob Tian, Yang Hermstrüwer, Yoan Huang, Yinya Akhtar, Mubashara Salimbeni, Etienne Geering, Florian Dreyer, Oliver Brunner, Daniel Leippold, Markus Sachan, Mrinmaya Stremitzer, Alexander Engel, Christoph Ash, Elliott Niklaus, Joel |
| contents | Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12864 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LEXam: Benchmarking Legal Reasoning on 340 Law Exams Fan, Yu Ni, Jingwei Merane, Jakob Tian, Yang Hermstrüwer, Yoan Huang, Yinya Akhtar, Mubashara Salimbeni, Etienne Geering, Florian Dreyer, Oliver Brunner, Daniel Leippold, Markus Sachan, Mrinmaya Stremitzer, Alexander Engel, Christoph Ash, Elliott Niklaus, Joel Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/. |
| title | LEXam: Benchmarking Legal Reasoning on 340 Law Exams |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 |
| url | https://arxiv.org/abs/2505.12864 |