On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929642546397184 |
|---|---|
| author | Wendlinger, Lorenz Braun, Christian Zubaer, Abdullah Al Nonn, Simon Alexander Großkopf, Sarah Fellicious, Christofer Granitzer, Michael |
| author_facet | Wendlinger, Lorenz Braun, Christian Zubaer, Abdullah Al Nonn, Simon Alexander Großkopf, Sarah Fellicious, Christofer Granitzer, Michael |
| contents | We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15902 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education Wendlinger, Lorenz Braun, Christian Zubaer, Abdullah Al Nonn, Simon Alexander Großkopf, Sarah Fellicious, Christofer Granitzer, Michael Computation and Language Artificial Intelligence We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case. |
| title | On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.15902 |