TransBench: Benchmarking Machine Translation for Industrial-Scale Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910957199949824 |
|---|---|
| author | Li, Haijun Shi, Tianqi Shang, Zifu Han, Yuxuan Zhao, Xueyu Wang, Hao Qian, Yu Qian, Zhiqiang Xu, Linlong Wu, Minghao Lyu, Chenyang Wang, Longyue Tang, Gongbo Luo, Weihua Xu, Zhao Zhang, Kaifu |
| author_facet | Li, Haijun Shi, Tianqi Shang, Zifu Han, Yuxuan Zhao, Xueyu Wang, Hao Qian, Yu Qian, Zhiqiang Xu, Linlong Wu, Minghao Lyu, Chenyang Wang, Longyue Tang, Gongbo Luo, Weihua Xu, Zhao Zhang, Kaifu |
| contents | Machine translation (MT) has become indispensable for cross-border communication in globalized industries like e-commerce, finance, and legal services, with recent advancements in large language models (LLMs) significantly enhancing translation quality. However, applying general-purpose MT models to industrial scenarios reveals critical limitations due to domain-specific terminology, cultural nuances, and stylistic conventions absent in generic benchmarks. Existing evaluation frameworks inadequately assess performance in specialized contexts, creating a gap between academic benchmarks and real-world efficacy. To address this, we propose a three-level translation capability framework: (1) Basic Linguistic Competence, (2) Domain-Specific Proficiency, and (3) Cultural Adaptation, emphasizing the need for holistic evaluation across these dimensions. We introduce TransBench, a benchmark tailored for industrial MT, initially targeting international e-commerce with 17,000 professionally translated sentences spanning 4 main scenarios and 33 language pairs. TransBench integrates traditional metrics (BLEU, TER) with Marco-MOS, a domain-specific evaluation model, and provides guidelines for reproducible benchmark construction. Our contributions include: (1) a structured framework for industrial MT evaluation, (2) the first publicly available benchmark for e-commerce translation, (3) novel metrics probing multi-level translation quality, and (4) open-sourced evaluation tools. This work bridges the evaluation gap, enabling researchers and practitioners to systematically assess and enhance MT systems for industry-specific needs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14244 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TransBench: Benchmarking Machine Translation for Industrial-Scale Applications Li, Haijun Shi, Tianqi Shang, Zifu Han, Yuxuan Zhao, Xueyu Wang, Hao Qian, Yu Qian, Zhiqiang Xu, Linlong Wu, Minghao Lyu, Chenyang Wang, Longyue Tang, Gongbo Luo, Weihua Xu, Zhao Zhang, Kaifu Computation and Language Machine translation (MT) has become indispensable for cross-border communication in globalized industries like e-commerce, finance, and legal services, with recent advancements in large language models (LLMs) significantly enhancing translation quality. However, applying general-purpose MT models to industrial scenarios reveals critical limitations due to domain-specific terminology, cultural nuances, and stylistic conventions absent in generic benchmarks. Existing evaluation frameworks inadequately assess performance in specialized contexts, creating a gap between academic benchmarks and real-world efficacy. To address this, we propose a three-level translation capability framework: (1) Basic Linguistic Competence, (2) Domain-Specific Proficiency, and (3) Cultural Adaptation, emphasizing the need for holistic evaluation across these dimensions. We introduce TransBench, a benchmark tailored for industrial MT, initially targeting international e-commerce with 17,000 professionally translated sentences spanning 4 main scenarios and 33 language pairs. TransBench integrates traditional metrics (BLEU, TER) with Marco-MOS, a domain-specific evaluation model, and provides guidelines for reproducible benchmark construction. Our contributions include: (1) a structured framework for industrial MT evaluation, (2) the first publicly available benchmark for e-commerce translation, (3) novel metrics probing multi-level translation quality, and (4) open-sourced evaluation tools. This work bridges the evaluation gap, enabling researchers and practitioners to systematically assess and enhance MT systems for industry-specific needs. |
| title | TransBench: Benchmarking Machine Translation for Industrial-Scale Applications |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.14244 |