Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.11938 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916107633295360 |
|---|---|
| author | Ruder, Sebastian Clark, Jonathan H. Gutkin, Alexander Kale, Mihir Ma, Min Nicosia, Massimo Rijhwani, Shruti Riley, Parker Sarr, Jean-Michel A. Wang, Xinyi Wieting, John Gupta, Nitish Katanova, Anna Kirov, Christo Dickinson, Dana L. Roark, Brian Samanta, Bidisha Tao, Connie Adelani, David I. Axelrod, Vera Caswell, Isaac Cherry, Colin Garrette, Dan Ingle, Reeve Johnson, Melvin Panteleev, Dmitry Talukdar, Partha |
| author_facet | Ruder, Sebastian Clark, Jonathan H. Gutkin, Alexander Kale, Mihir Ma, Min Nicosia, Massimo Rijhwani, Shruti Riley, Parker Sarr, Jean-Michel A. Wang, Xinyi Wieting, John Gupta, Nitish Katanova, Anna Kirov, Christo Dickinson, Dana L. Roark, Brian Samanta, Bidisha Tao, Connie Adelani, David I. Axelrod, Vera Caswell, Isaac Cherry, Colin Garrette, Dan Ingle, Reeve Johnson, Melvin Panteleev, Dmitry Talukdar, Partha |
| contents | Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) -- languages for which NLP re-search is particularly far behind in meeting user needs -- it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text-only, multi-modal (vision, audio, and text),supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_11938 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages Ruder, Sebastian Clark, Jonathan H. Gutkin, Alexander Kale, Mihir Ma, Min Nicosia, Massimo Rijhwani, Shruti Riley, Parker Sarr, Jean-Michel A. Wang, Xinyi Wieting, John Gupta, Nitish Katanova, Anna Kirov, Christo Dickinson, Dana L. Roark, Brian Samanta, Bidisha Tao, Connie Adelani, David I. Axelrod, Vera Caswell, Isaac Cherry, Colin Garrette, Dan Ingle, Reeve Johnson, Melvin Panteleev, Dmitry Talukdar, Partha Computation and Language Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) -- languages for which NLP re-search is particularly far behind in meeting user needs -- it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text-only, multi-modal (vision, audio, and text),supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models |
| title | XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2305.11938 |