SMOL: Professionally translated parallel data for 115 under-represented languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911241756213248 |
|---|---|
| author | Caswell, Isaac Nielsen, Elizabeth Luo, Jiaming Cherry, Colin Kovacs, Geza Shemtov, Hadar Talukdar, Partha Tewari, Dinesh Diane, Baba Mamadi Diane, Djibrila Cissé, Solo Farabado Doumbouya, Koulako Moussa Ferrante, Edoardo Guasoni, Alessandro Homan, Christopher Keita, Mamadou K. DebBarma, Sudhamoy Kuzhuget, Ali Anugraha, David Habibi, Muhammad Ravi Shulthan Winata, Genta Indra Munthali, Anthony Ahmadi, Sina Chemyshev, Andrei Lau, Mingfei Eng, Jonathan |
| author_facet | Caswell, Isaac Nielsen, Elizabeth Luo, Jiaming Cherry, Colin Kovacs, Geza Shemtov, Hadar Talukdar, Partha Tewari, Dinesh Diane, Baba Mamadi Diane, Djibrila Cissé, Solo Farabado Doumbouya, Koulako Moussa Ferrante, Edoardo Guasoni, Alessandro Homan, Christopher Keita, Mamadou K. DebBarma, Sudhamoy Kuzhuget, Ali Anugraha, David Habibi, Muhammad Ravi Shulthan Winata, Genta Indra Munthali, Anthony Ahmadi, Sina Chemyshev, Andrei Lau, Mingfei Eng, Jonathan |
| contents | We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock machine translation for low-resource languages. SMOL has been translated into 124 (and growing) under-resourced languages (125 language pairs), including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOLSENT, a set of sentences chosen for broad unique token coverage, and SMOLDOC, a document-level resource focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust chrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOLDOC, yielding the first factuality datasets for most of these languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12301 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SMOL: Professionally translated parallel data for 115 under-represented languages Caswell, Isaac Nielsen, Elizabeth Luo, Jiaming Cherry, Colin Kovacs, Geza Shemtov, Hadar Talukdar, Partha Tewari, Dinesh Diane, Baba Mamadi Diane, Djibrila Cissé, Solo Farabado Doumbouya, Koulako Moussa Ferrante, Edoardo Guasoni, Alessandro Homan, Christopher Keita, Mamadou K. DebBarma, Sudhamoy Kuzhuget, Ali Anugraha, David Habibi, Muhammad Ravi Shulthan Winata, Genta Indra Munthali, Anthony Ahmadi, Sina Chemyshev, Andrei Lau, Mingfei Eng, Jonathan Computation and Language We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock machine translation for low-resource languages. SMOL has been translated into 124 (and growing) under-resourced languages (125 language pairs), including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOLSENT, a set of sentences chosen for broad unique token coverage, and SMOLDOC, a document-level resource focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust chrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOLDOC, yielding the first factuality datasets for most of these languages. |
| title | SMOL: Professionally translated parallel data for 115 under-represented languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12301 |