Cover Image

OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Servantez, Sergio, Lawsky, Sarah B., Jain, Rajiv, Linna Jr., Daniel W., Hammond, Kristian
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Chain of Logic: Rule-Based Reasoning with Large Language Models
by: Servantez, Sergio, et al.
Published: (2024)

A Reasoning-Focused Legal Retrieval Benchmark
by: Zheng, Lucia, et al.
Published: (2025)

LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks
by: Fujita, Shogo, et al.
Published: (2025)

ALARB: An Arabic Legal Argument Reasoning Benchmark
by: Shairah, Harethah Abu, et al.
Published: (2025)

Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities
by: Oh, Hongseok, et al.
Published: (2025)

Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
by: Khatri, Mann, et al.
Published: (2025)

Challenges for Generative AI in Legal Reasoning
by: Linna, Eljas, et al.
Published: (2025)

LEXam: Benchmarking Legal Reasoning on 340 Law Exams
by: Fan, Yu, et al.
Published: (2025)

MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs
by: Bouchekif, Abdessalam, et al.
Published: (2026)

AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction
by: Li, Yifei, et al.
Published: (2026)

IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning
by: Joshi, Abhinav, et al.
Published: (2024)

MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoning
by: Jing, Huihao, et al.
Published: (2025)

CSCBench: A PVC Diagnostic Benchmark for Commodity Supply Chain Reasoning
by: Cui, Yaxin, et al.
Published: (2026)

DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
by: Zhu, Yakun, et al.
Published: (2025)

OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
by: Li, Yanhong, et al.
Published: (2025)

Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents
by: Li, Yaocong, et al.
Published: (2026)

Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases
by: AlDahoul, Nouar, et al.
Published: (2025)

Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks
by: Wang, Xinhe, et al.
Published: (2025)

Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys
by: Afane, Mohamed, et al.
Published: (2026)

GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations
by: Chlapanis, Odysseas S., et al.
Published: (2025)

BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefs
by: Woo, Jesse, et al.
Published: (2025)

LegalCore: A Dataset for Event Coreference Resolution in Legal Documents
by: Wei, Kangda, et al.
Published: (2025)

A Benchmark Dataset and Evaluation Framework for Vietnamese Large Language Models in Customer Support
by: Nguyen, Long S. T., et al.
Published: (2025)

One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
by: Stern, Ronja, et al.
Published: (2023)

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
by: Nagl, Sebastian, et al.
Published: (2026)

VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models
by: Dong, Nguyen Tien, et al.
Published: (2025)

UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
by: Ovcharov, Volodymyr
Published: (2026)

Benchmarking and Learning Real-World Customer Service Dialogue
by: Gao, Tianhong, et al.
Published: (2025)

PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian
by: Mozafari, Jamshid, et al.
Published: (2026)

Open Shouldn't Mean Exempt: Open-Source Exceptionalism and Generative AI
by: Atkinson, David
Published: (2025)

The Massive Legal Embedding Benchmark (MLEB)
by: Butler, Umar, et al.
Published: (2025)

LegalBench.PT: A Benchmark for Portuguese Law
by: Canaverde, Beatriz, et al.
Published: (2025)

PILOT-Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC-Aligned Classification Tasks
by: Jang, Yehoon, et al.
Published: (2026)

RingSQL: Generating Synthetic Data with Schema-Independent Templates for Text-to-SQL Reasoning Models
by: Sterbentz, Marko, et al.
Published: (2026)

CharacterBench: Benchmarking Character Customization of Large Language Models
by: Zhou, Jinfeng, et al.
Published: (2024)

CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis
by: Xu, Xinzhe, et al.
Published: (2025)

OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs
by: Wang, Yuxia, et al.
Published: (2024)

Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
by: Hagendorff, Thilo, et al.
Published: (2025)

LexTime: A Benchmark for Temporal Ordering of Legal Events
by: Barale, Claire, et al.
Published: (2025)

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks
by: Xie, Yiqing, et al.
Published: (2024)