Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2509.17249 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911168080117760 |
|---|---|
| author | Wang, Kuang-Da Ding, Shuoyang Yang, Chao-Han Huck Hsieh, Ping-Chun Peng, Wen-Chih Lavrukhin, Vitaly Ginsburg, Boris |
| author_facet | Wang, Kuang-Da Ding, Shuoyang Yang, Chao-Han Huck Hsieh, Ping-Chun Peng, Wen-Chih Lavrukhin, Vitaly Ginsburg, Boris |
| contents | Despite Large Language Models (LLMs) demonstrating superior translation performance and long-context capabilities, evaluation methodologies remain constrained to sentence-level assessment due to dataset limitations, token number restrictions in metrics, and rigid sentence boundary requirements. We introduce SEGALE, an evaluation scheme that extends existing automatic metrics to long-document translation by treating documents as continuous text and applying sentence segmentation and alignment methods. Our approach enables previously unattainable document-level evaluation, handling translations of arbitrary length generated with document-level prompts while accounting for under-/over-translations and varied sentence boundaries. Experiments show our scheme significantly outperforms existing long-form document evaluation schemes, while being comparable to evaluations performed with groundtruth sentence alignments. Additionally, we apply our scheme to book-length texts and newly demonstrate that many open-weight LLMs fail to effectively translate documents at their reported maximum context lengths. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17249 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Extending Automatic Machine Translation Evaluation to Book-Length Documents Wang, Kuang-Da Ding, Shuoyang Yang, Chao-Han Huck Hsieh, Ping-Chun Peng, Wen-Chih Lavrukhin, Vitaly Ginsburg, Boris Computation and Language Despite Large Language Models (LLMs) demonstrating superior translation performance and long-context capabilities, evaluation methodologies remain constrained to sentence-level assessment due to dataset limitations, token number restrictions in metrics, and rigid sentence boundary requirements. We introduce SEGALE, an evaluation scheme that extends existing automatic metrics to long-document translation by treating documents as continuous text and applying sentence segmentation and alignment methods. Our approach enables previously unattainable document-level evaluation, handling translations of arbitrary length generated with document-level prompts while accounting for under-/over-translations and varied sentence boundaries. Experiments show our scheme significantly outperforms existing long-form document evaluation schemes, while being comparable to evaluations performed with groundtruth sentence alignments. Additionally, we apply our scheme to book-length texts and newly demonstrate that many open-weight LLMs fail to effectively translate documents at their reported maximum context lengths. |
| title | Extending Automatic Machine Translation Evaluation to Book-Length Documents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.17249 |