Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929736864759808 |
|---|---|
| author | Wu, Junchao Zhan, Runzhe Wong, Derek F. Yang, Shu Liu, Xuebo Chao, Lidia S. Zhang, Min |
| author_facet | Wu, Junchao Zhan, Runzhe Wong, Derek F. Yang, Shu Liu, Xuebo Chao, Lidia S. Zhang, Min |
| contents | The efficacy of detectors for texts generated by large language models (LLMs) substantially depends on the availability of large-scale training data. However, white-box zero-shot detectors, which require no such data, are limited by the accessibility of the source model of the LLM-generated text. In this paper, we propose a simple yet effective black-box zero-shot detection approach based on the observation that, from the perspective of LLMs, human-written texts typically contain more grammatical errors than LLM-generated texts. This approach involves calculating the Grammar Error Correction Score (GECScore) for the given text to differentiate between human-written and LLM-generated text. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods, achieving an average AUROC of 98.62% across XSum and Writing Prompts dataset. Additionally, our approach demonstrates strong reliability in the wild, exhibiting robust generalization and resistance to paraphrasing attacks. Data and code are available at: https://github.com/NLP2CT/GECScore. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04286 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore Wu, Junchao Zhan, Runzhe Wong, Derek F. Yang, Shu Liu, Xuebo Chao, Lidia S. Zhang, Min Computation and Language The efficacy of detectors for texts generated by large language models (LLMs) substantially depends on the availability of large-scale training data. However, white-box zero-shot detectors, which require no such data, are limited by the accessibility of the source model of the LLM-generated text. In this paper, we propose a simple yet effective black-box zero-shot detection approach based on the observation that, from the perspective of LLMs, human-written texts typically contain more grammatical errors than LLM-generated texts. This approach involves calculating the Grammar Error Correction Score (GECScore) for the given text to differentiate between human-written and LLM-generated text. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods, achieving an average AUROC of 98.62% across XSum and Writing Prompts dataset. Additionally, our approach demonstrates strong reliability in the wild, exhibiting robust generalization and resistance to paraphrasing attacks. Data and code are available at: https://github.com/NLP2CT/GECScore. |
| title | Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.04286 |