MultiLegalPile: A 689GB Multilingual Legal Corpus
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911879957315584 |
|---|---|
| author | Niklaus, Joel Matoshi, Veton Stürmer, Matthias Chalkidis, Ilias Ho, Daniel E. |
| author_facet | Niklaus, Joel Matoshi, Veton Stürmer, Matthias Chalkidis, Ilias Ho, Daniel E. |
| contents | Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for the English language. We curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. The MultiLegalPile corpus, which includes diverse legal data sources with varying licenses, allows for pretraining NLP models under fair use, with more permissive licenses for the Eurlex Resources and Legal mC4 subsets. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, the trained models, and all of the code under the most open possible licenses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_02069 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MultiLegalPile: A 689GB Multilingual Legal Corpus Niklaus, Joel Matoshi, Veton Stürmer, Matthias Chalkidis, Ilias Ho, Daniel E. Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for the English language. We curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. The MultiLegalPile corpus, which includes diverse legal data sources with varying licenses, allows for pretraining NLP models under fair use, with more permissive licenses for the Eurlex Resources and Legal mC4 subsets. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, the trained models, and all of the code under the most open possible licenses. |
| title | MultiLegalPile: A 689GB Multilingual Legal Corpus |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 |
| url | https://arxiv.org/abs/2306.02069 |