A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916946822299648 |
|---|---|
| author | Shafayat, Sheikh Yoon, Dongkeun Jang, Woori Choi, Jiwoo Oh, Alice Jung, Seohyon |
| author_facet | Shafayat, Sheikh Yoon, Dongkeun Jang, Woori Choi, Jiwoo Oh, Alice Jung, Seohyon |
| contents | In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01340 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls Shafayat, Sheikh Yoon, Dongkeun Jang, Woori Choi, Jiwoo Oh, Alice Jung, Seohyon Computation and Language In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works. |
| title | A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.01340 |