Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913498359922688 |
|---|---|
| author | Li, Jonathan Bhambhoria, Rohan Dahan, Samuel Zhu, Xiaodan |
| author_facet | Li, Jonathan Bhambhoria, Rohan Dahan, Samuel Zhu, Xiaodan |
| contents | Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07713 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice Li, Jonathan Bhambhoria, Rohan Dahan, Samuel Zhu, Xiaodan Computation and Language Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models. |
| title | Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.07713 |