Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910176769998848 |
|---|---|
| author | Wang, Yuxia Xing, Rui Mansurov, Jonibek Puccetti, Giovanni Xie, Zhuohan Ta, Minh Ngoc Geng, Jiahui Su, Jinyan Abassy, Mervat Ahmed, Saad El Dine Elozeiri, Kareem Laiyk, Nurkhan Goloburda, Maiya Mahmoud, Tarek Tomar, Raj Vardhan Aziz, Alexander Koike, Ryuto Kaneko, Masahiro Shelmanov, Artem Artemova, Ekaterina Mikhailov, Vladislav Tsvigun, Akim Aji, Alham Fikri Habash, Nizar Gurevych, Iryna Nakov, Preslav |
| author_facet | Wang, Yuxia Xing, Rui Mansurov, Jonibek Puccetti, Giovanni Xie, Zhuohan Ta, Minh Ngoc Geng, Jiahui Su, Jinyan Abassy, Mervat Ahmed, Saad El Dine Elozeiri, Kareem Laiyk, Nurkhan Goloburda, Maiya Mahmoud, Tarek Tomar, Raj Vardhan Aziz, Alexander Koike, Ryuto Kaneko, Masahiro Shelmanov, Artem Artemova, Ekaterina Mikhailov, Vladislav Tsvigun, Akim Aji, Alham Fikri Habash, Nizar Gurevych, Iryna Nakov, Preslav |
| contents | Prior studies have shown that distinguishing text generated by Large Language Models (LLMs) from human-written one is highly challenging for humans, and often no better than random guessing. To verify the generalizability of this finding across languages and domains, we perform an extensive case study to identify the upper bound of human detection accuracy. Across 16 datasets covering 9 languages and 9 domains, 19 annotators achieved an average detection accuracy of 87.6%, thus challenging previous conclusions. We find that major gaps between human and machine text lie in concreteness, cultural nuances, and diversity. Prompting by explicitly explaining the distinctions in the prompts can partially bridge the gaps in over 50% of the cases. However, we also find that humans do not always prefer human-written text, particularly when they cannot clearly identify its source. We release our dataset, the human labels, and the annotator metadata at https://github.com/xnlp-lab/HumanEval-MGT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11614 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI Wang, Yuxia Xing, Rui Mansurov, Jonibek Puccetti, Giovanni Xie, Zhuohan Ta, Minh Ngoc Geng, Jiahui Su, Jinyan Abassy, Mervat Ahmed, Saad El Dine Elozeiri, Kareem Laiyk, Nurkhan Goloburda, Maiya Mahmoud, Tarek Tomar, Raj Vardhan Aziz, Alexander Koike, Ryuto Kaneko, Masahiro Shelmanov, Artem Artemova, Ekaterina Mikhailov, Vladislav Tsvigun, Akim Aji, Alham Fikri Habash, Nizar Gurevych, Iryna Nakov, Preslav Computation and Language Artificial Intelligence Prior studies have shown that distinguishing text generated by Large Language Models (LLMs) from human-written one is highly challenging for humans, and often no better than random guessing. To verify the generalizability of this finding across languages and domains, we perform an extensive case study to identify the upper bound of human detection accuracy. Across 16 datasets covering 9 languages and 9 domains, 19 annotators achieved an average detection accuracy of 87.6%, thus challenging previous conclusions. We find that major gaps between human and machine text lie in concreteness, cultural nuances, and diversity. Prompting by explicitly explaining the distinctions in the prompts can partially bridge the gaps in over 50% of the cases. However, we also find that humans do not always prefer human-written text, particularly when they cannot clearly identify its source. We release our dataset, the human labels, and the annotator metadata at https://github.com/xnlp-lab/HumanEval-MGT. |
| title | Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.11614 |