Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.00686 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929261548404736 |
|---|---|
| author | Arnett, Catherine Chang, Tyler A. Bergen, Benjamin K. |
| author_facet | Arnett, Catherine Chang, Tyler A. Bergen, Benjamin K. |
| contents | How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages Arnett, Catherine Chang, Tyler A. Bergen, Benjamin K. Computation and Language How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices. |
| title | A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.00686 |