Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12743 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913908829192192 |
|---|---|
| author | Ji, Jiazhou Guo, Jie Qiu, Weidong Huang, Zheng Xu, Yang Lu, Xinru Jiang, Xiaoyu Li, Ruizhe Li, Shujun |
| author_facet | Ji, Jiazhou Guo, Jie Qiu, Weidong Huang, Zheng Xu, Yang Lu, Xinru Jiang, Xiaoyu Li, Ruizhe Li, Shujun |
| contents | Distinguishing between human- and LLM-generated texts is crucial given the risks associated with misuse of LLMs. This paper investigates detection and explanation capabilities of current LLMs across two settings: binary (human vs. LLM-generated) and ternary classification (including an ``undecided'' class). We evaluate 6 close- and open-source LLMs of varying sizes and find that self-detection (LLMs identifying their own outputs) consistently outperforms cross-detection (identifying outputs from other LLMs), though both remain suboptimal. Introducing a ternary classification framework improves both detection accuracy and explanation quality across all models. Through comprehensive quantitative and qualitative analyses using our human-annotated dataset, we identify key explanation failures, primarily reliance on inaccurate features, hallucinations, and flawed reasoning. Our findings underscore the limitations of current LLMs in self-detection and self-explanation, highlighting the need for further research to address overfitting and enhance generalizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12743 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | "I know myself better, but not really greatly": How Well Can LLMs Detect and Explain LLM-Generated Texts? Ji, Jiazhou Guo, Jie Qiu, Weidong Huang, Zheng Xu, Yang Lu, Xinru Jiang, Xiaoyu Li, Ruizhe Li, Shujun Computation and Language Artificial Intelligence Distinguishing between human- and LLM-generated texts is crucial given the risks associated with misuse of LLMs. This paper investigates detection and explanation capabilities of current LLMs across two settings: binary (human vs. LLM-generated) and ternary classification (including an ``undecided'' class). We evaluate 6 close- and open-source LLMs of varying sizes and find that self-detection (LLMs identifying their own outputs) consistently outperforms cross-detection (identifying outputs from other LLMs), though both remain suboptimal. Introducing a ternary classification framework improves both detection accuracy and explanation quality across all models. Through comprehensive quantitative and qualitative analyses using our human-annotated dataset, we identify key explanation failures, primarily reliance on inaccurate features, hallucinations, and flawed reasoning. Our findings underscore the limitations of current LLMs in self-detection and self-explanation, highlighting the need for further research to address overfitting and enhance generalizability. |
| title | "I know myself better, but not really greatly": How Well Can LLMs Detect and Explain LLM-Generated Texts? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.12743 |