Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908782937767936 |
|---|---|
| author | Jia, Zheng Yue, Shengbin Chen, Wei Wang, Siyuan Liu, Yidong Li, Zejun Song, Yun Wei, Zhongyu |
| author_facet | Jia, Zheng Yue, Shengbin Chen, Wei Wang, Siyuan Liu, Yidong Li, Zejun Song, Yun Wei, Zhongyu |
| contents | The gap between static benchmarks and the dynamic nature of real-world legal practice poses a key barrier to advancing legal intelligence. To this end, we introduce J1-ENVS, the first interactive and dynamic legal environment tailored for LLM-based agents. Guided by legal experts, it comprises six representative scenarios from Chinese legal practices across three levels of environmental complexity. We further introduce J1-EVAL, a fine-grained evaluation framework, designed to assess both task performance and procedural compliance across varying levels of legal proficiency. Extensive experiments on 17 LLM agents reveal that, while many models demonstrate solid legal knowledge, they struggle with procedural execution in dynamic settings. Even the SOTA model, GPT-4o, falls short of 60% overall performance. These findings highlight persistent challenges in achieving dynamic legal intelligence and offer valuable insights to guide future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04037 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments Jia, Zheng Yue, Shengbin Chen, Wei Wang, Siyuan Liu, Yidong Li, Zejun Song, Yun Wei, Zhongyu Artificial Intelligence The gap between static benchmarks and the dynamic nature of real-world legal practice poses a key barrier to advancing legal intelligence. To this end, we introduce J1-ENVS, the first interactive and dynamic legal environment tailored for LLM-based agents. Guided by legal experts, it comprises six representative scenarios from Chinese legal practices across three levels of environmental complexity. We further introduce J1-EVAL, a fine-grained evaluation framework, designed to assess both task performance and procedural compliance across varying levels of legal proficiency. Extensive experiments on 17 LLM agents reveal that, while many models demonstrate solid legal knowledge, they struggle with procedural execution in dynamic settings. Even the SOTA model, GPT-4o, falls short of 60% overall performance. These findings highlight persistent challenges in achieving dynamic legal intelligence and offer valuable insights to guide future research. |
| title | Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.04037 |