The Factuality of Large Language Models in the Legal Domain
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912033610399744 |
|---|---|
| author | Hamdani, Rajaa El Bonald, Thomas Malliaros, Fragkiskos Holzenberger, Nils Suchanek, Fabian |
| author_facet | Hamdani, Rajaa El Bonald, Thomas Malliaros, Fragkiskos Holzenberger, Nils Suchanek, Fabian |
| contents | This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11798 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Factuality of Large Language Models in the Legal Domain Hamdani, Rajaa El Bonald, Thomas Malliaros, Fragkiskos Holzenberger, Nils Suchanek, Fabian Computation and Language Artificial Intelligence Information Retrieval Machine Learning This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%. |
| title | The Factuality of Large Language Models in the Legal Domain |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2409.11798 |