MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916279806328832 |
|---|---|
| author | Macko, Dominik Moro, Robert Uchendu, Adaku Lucas, Jason Samuel Yamashita, Michiharu Pikuliak, Matúš Srba, Ivan Le, Thai Lee, Dongwon Simko, Jakub Bielikova, Maria |
| author_facet | Macko, Dominik Moro, Robert Uchendu, Adaku Lucas, Jason Samuel Yamashita, Michiharu Pikuliak, Matúš Srba, Ivan Le, Thai Lee, Dongwon Simko, Jakub Bielikova, Maria |
| contents | There is a lack of research into capabilities of recent LLMs to generate convincing text in languages other than English and into performance of detectors of machine-generated text in multilingual settings. This is also reflected in the available benchmarks which lack authentic texts in languages other than English and predominantly cover older generators. To fill this gap, we introduce MULTITuDE, a novel benchmarking dataset for multilingual machine-generated text detection comprising of 74,081 authentic and machine-generated texts in 11 languages (ar, ca, cs, de, en, es, nl, pt, ru, uk, and zh) generated by 8 multilingual LLMs. Using this benchmark, we compare the performance of zero-shot (statistical and black-box) and fine-tuned detectors. Considering the multilinguality, we evaluate 1) how these detectors generalize to unseen languages (linguistically similar as well as dissimilar) and unseen LLMs and 2) whether the detectors improve their performance when trained on multiple languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13606 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark Macko, Dominik Moro, Robert Uchendu, Adaku Lucas, Jason Samuel Yamashita, Michiharu Pikuliak, Matúš Srba, Ivan Le, Thai Lee, Dongwon Simko, Jakub Bielikova, Maria Computation and Language Artificial Intelligence There is a lack of research into capabilities of recent LLMs to generate convincing text in languages other than English and into performance of detectors of machine-generated text in multilingual settings. This is also reflected in the available benchmarks which lack authentic texts in languages other than English and predominantly cover older generators. To fill this gap, we introduce MULTITuDE, a novel benchmarking dataset for multilingual machine-generated text detection comprising of 74,081 authentic and machine-generated texts in 11 languages (ar, ca, cs, de, en, es, nl, pt, ru, uk, and zh) generated by 8 multilingual LLMs. Using this benchmark, we compare the performance of zero-shot (statistical and black-box) and fine-tuned detectors. Considering the multilinguality, we evaluate 1) how these detectors generalize to unseen languages (linguistically similar as well as dissimilar) and unseen LLMs and 2) whether the detectors improve their performance when trained on multiple languages. |
| title | MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2310.13606 |