UPRPRC: Unified Pipeline for Reproducing Parallel Resources -- Corpus from the United Nations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918144299237376 |
|---|---|
| author | Lu, Qiuyang Shen, Fangjian Tang, Zhengkai Liu, Qiang Cheng, Hexuan Liu, Hui Wen, Wushao |
| author_facet | Lu, Qiuyang Shen, Fangjian Tang, Zhengkai Liu, Qiang Cheng, Hexuan Liu, Hui Wen, Wushao |
| contents | The quality and accessibility of multilingual datasets are crucial for advancing machine translation. However, previous corpora built from United Nations documents have suffered from issues such as opaque process, difficulty of reproduction, and limited scale. To address these challenges, we introduce a complete end-to-end solution, from data acquisition via web scraping to text alignment. The entire process is fully reproducible, with a minimalist single-machine example and optional distributed computing steps for scalability. At its core, we propose a new Graph-Aided Paragraph Alignment (GAPA) algorithm for efficient and flexible paragraph-level alignment. The resulting corpus contains over 713 million English tokens, more than doubling the scale of prior work. To the best of our knowledge, this represents the largest publicly available parallel corpus composed entirely of human-translated, non-AI-generated content. Our code and corpus are accessible under the MIT License. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15789 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UPRPRC: Unified Pipeline for Reproducing Parallel Resources -- Corpus from the United Nations Lu, Qiuyang Shen, Fangjian Tang, Zhengkai Liu, Qiang Cheng, Hexuan Liu, Hui Wen, Wushao Computation and Language Machine Learning The quality and accessibility of multilingual datasets are crucial for advancing machine translation. However, previous corpora built from United Nations documents have suffered from issues such as opaque process, difficulty of reproduction, and limited scale. To address these challenges, we introduce a complete end-to-end solution, from data acquisition via web scraping to text alignment. The entire process is fully reproducible, with a minimalist single-machine example and optional distributed computing steps for scalability. At its core, we propose a new Graph-Aided Paragraph Alignment (GAPA) algorithm for efficient and flexible paragraph-level alignment. The resulting corpus contains over 713 million English tokens, more than doubling the scale of prior work. To the best of our knowledge, this represents the largest publicly available parallel corpus composed entirely of human-translated, non-AI-generated content. Our code and corpus are accessible under the MIT License. |
| title | UPRPRC: Unified Pipeline for Reproducing Parallel Resources -- Corpus from the United Nations |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.15789 |