SwiLTra-Bench: The Swiss Legal Translation Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918039736287232 |
|---|---|
| author | Niklaus, Joel Merane, Jakob Nenadic, Luka Ahmadi, Sina Gao, Yingqiang Chevalley, Cyrill A. H. Humbel, Claude Gösken, Christophe Tanzi, Lorenzo Lüthi, Thomas Palombo, Stefan Poff, Spencer Yang, Boling Wu, Nan Guillod, Matthew Mamié, Robin Brunner, Daniel Pereyra, Julio Grupen, Niko |
| author_facet | Niklaus, Joel Merane, Jakob Nenadic, Luka Ahmadi, Sina Gao, Yingqiang Chevalley, Cyrill A. H. Humbel, Claude Gösken, Christophe Tanzi, Lorenzo Lüthi, Thomas Palombo, Stefan Poff, Spencer Yang, Boling Wu, Nan Guillod, Matthew Mamié, Robin Brunner, Daniel Pereyra, Julio Grupen, Niko |
| contents | In Switzerland legal translation is uniquely important due to the country's four official languages and requirements for multilingual legal documentation. However, this process traditionally relies on professionals who must be both legal experts and skilled translators -- creating bottlenecks and impacting effective access to justice. To address this challenge, we introduce SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. Our systematic evaluation reveals that frontier models achieve superior translation performance across all document types, while specialized translation systems excel specifically in laws but under-perform in headnotes. Through rigorous testing and human expert validation, we demonstrate that while fine-tuning open SLMs significantly improves their translation quality, they still lag behind the best zero-shot prompted frontier models such as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM evaluation system that aligns best with human expert assessments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01372 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SwiLTra-Bench: The Swiss Legal Translation Benchmark Niklaus, Joel Merane, Jakob Nenadic, Luka Ahmadi, Sina Gao, Yingqiang Chevalley, Cyrill A. H. Humbel, Claude Gösken, Christophe Tanzi, Lorenzo Lüthi, Thomas Palombo, Stefan Poff, Spencer Yang, Boling Wu, Nan Guillod, Matthew Mamié, Robin Brunner, Daniel Pereyra, Julio Grupen, Niko Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 In Switzerland legal translation is uniquely important due to the country's four official languages and requirements for multilingual legal documentation. However, this process traditionally relies on professionals who must be both legal experts and skilled translators -- creating bottlenecks and impacting effective access to justice. To address this challenge, we introduce SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. Our systematic evaluation reveals that frontier models achieve superior translation performance across all document types, while specialized translation systems excel specifically in laws but under-perform in headnotes. Through rigorous testing and human expert validation, we demonstrate that while fine-tuning open SLMs significantly improves their translation quality, they still lag behind the best zero-shot prompted frontier models such as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM evaluation system that aligns best with human expert assessments. |
| title | SwiLTra-Bench: The Swiss Legal Translation Benchmark |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T50 I.2 |
| url | https://arxiv.org/abs/2503.01372 |