Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.15044 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913919732285440 |
|---|---|
| author | Li, Haoyi Yuan, Angela Yifei Han, Soyeon Caren Leckie, Christopher |
| author_facet | Li, Haoyi Yuan, Angela Yifei Han, Soyeon Caren Leckie, Christopher |
| contents | The increasing capability of large language models (LLMs) to generate synthetic content has heightened concerns about their misuse, driving the development of Machine-Generated Text (MGT) detection models. However, these detectors face significant challenges due to the lack of high-quality synthetic datasets for training. To address this issue, we propose SPADE, a structured framework for detecting synthetic dialogues using prompt-based positive and negative samples. Our proposed methods yield 14 new dialogue datasets, which we benchmark against eight MGT detection models. The results demonstrate improved generalization performance when utilizing a mixed dataset produced by proposed augmentation frameworks, offering a practical approach to enhancing LLM application security. Considering that real-world agents lack knowledge of future opponent utterances, we simulate online dialogue detection and examine the relationship between chat history length and detection accuracy. Our open-source datasets, code and prompts can be downloaded from https://github.com/AngieYYF/SPADE-customer-service-dialogue. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_15044 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection Li, Haoyi Yuan, Angela Yifei Han, Soyeon Caren Leckie, Christopher Computation and Language The increasing capability of large language models (LLMs) to generate synthetic content has heightened concerns about their misuse, driving the development of Machine-Generated Text (MGT) detection models. However, these detectors face significant challenges due to the lack of high-quality synthetic datasets for training. To address this issue, we propose SPADE, a structured framework for detecting synthetic dialogues using prompt-based positive and negative samples. Our proposed methods yield 14 new dialogue datasets, which we benchmark against eight MGT detection models. The results demonstrate improved generalization performance when utilizing a mixed dataset produced by proposed augmentation frameworks, offering a practical approach to enhancing LLM application security. Considering that real-world agents lack knowledge of future opponent utterances, we simulate online dialogue detection and examine the relationship between chat history length and detection accuracy. Our open-source datasets, code and prompts can be downloaded from https://github.com/AngieYYF/SPADE-customer-service-dialogue. |
| title | SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.15044 |