An Information-Theoretic Approach for Detecting Edits in AI-Generated Text
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912000268828672 |
|---|---|
| author | Kashtan, Idan Kipnis, Alon |
| author_facet | Kashtan, Idan Kipnis, Alon |
| contents | We propose a method to determine whether a given article was written entirely by a generative language model or perhaps contains edits by a different author, possibly a human. Our process involves multiple tests for the origin of individual sentences or other pieces of text and combining these tests using a method that is sensitive to rare alternatives, i.e., non-null effects are few and scattered across the text in unknown locations. Interestingly, this method also identifies pieces of text suspected to contain edits. We demonstrate the effectiveness of the method in detecting edits through extensive evaluations using real data and provide an information-theoretic analysis of the factors affecting its success. In particular, we discuss optimality properties under a theoretical framework for text editing saying that sentences are generated mainly by the language model, except perhaps for a few sentences that might have originated via a different mechanism. Our analysis raises several interesting research questions at the intersection of information theory and data science. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_12747 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | An Information-Theoretic Approach for Detecting Edits in AI-Generated Text Kashtan, Idan Kipnis, Alon Information Theory Artificial Intelligence Applications 62P25, 62H15, 68P30, 94A15, 68T01 I.2.7 We propose a method to determine whether a given article was written entirely by a generative language model or perhaps contains edits by a different author, possibly a human. Our process involves multiple tests for the origin of individual sentences or other pieces of text and combining these tests using a method that is sensitive to rare alternatives, i.e., non-null effects are few and scattered across the text in unknown locations. Interestingly, this method also identifies pieces of text suspected to contain edits. We demonstrate the effectiveness of the method in detecting edits through extensive evaluations using real data and provide an information-theoretic analysis of the factors affecting its success. In particular, we discuss optimality properties under a theoretical framework for text editing saying that sentences are generated mainly by the language model, except perhaps for a few sentences that might have originated via a different mechanism. Our analysis raises several interesting research questions at the intersection of information theory and data science. |
| title | An Information-Theoretic Approach for Detecting Edits in AI-Generated Text |
| topic | Information Theory Artificial Intelligence Applications 62P25, 62H15, 68P30, 94A15, 68T01 I.2.7 |
| url | https://arxiv.org/abs/2308.12747 |