Beyond Turing Test: Can GPT-4 Sway Experts' Decisions?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915032045977600 |
|---|---|
| author | Takayanagi, Takehiro Takamura, Hiroya Izumi, Kiyoshi Chen, Chung-Chi |
| author_facet | Takayanagi, Takehiro Takamura, Hiroya Izumi, Kiyoshi Chen, Chung-Chi |
| contents | In the post-Turing era, evaluating large language models (LLMs) involves assessing generated text based on readers' reactions rather than merely its indistinguishability from human-produced content. This paper explores how LLM-generated text impacts readers' decisions, focusing on both amateur and expert audiences. Our findings indicate that GPT-4 can generate persuasive analyses affecting the decisions of both amateurs and professionals. Furthermore, we evaluate the generated text from the aspects of grammar, convincingness, logical coherence, and usefulness. The results highlight a high correlation between real-world evaluation through audience reactions and the current multi-dimensional evaluators commonly used for generative models. Overall, this paper shows the potential and risk of using generated text to sway human decisions and also points out a new direction for evaluating generated text, i.e., leveraging the reactions and decisions of readers. We release our dataset to assist future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16710 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Turing Test: Can GPT-4 Sway Experts' Decisions? Takayanagi, Takehiro Takamura, Hiroya Izumi, Kiyoshi Chen, Chung-Chi Computational Engineering, Finance, and Science Computation and Language In the post-Turing era, evaluating large language models (LLMs) involves assessing generated text based on readers' reactions rather than merely its indistinguishability from human-produced content. This paper explores how LLM-generated text impacts readers' decisions, focusing on both amateur and expert audiences. Our findings indicate that GPT-4 can generate persuasive analyses affecting the decisions of both amateurs and professionals. Furthermore, we evaluate the generated text from the aspects of grammar, convincingness, logical coherence, and usefulness. The results highlight a high correlation between real-world evaluation through audience reactions and the current multi-dimensional evaluators commonly used for generative models. Overall, this paper shows the potential and risk of using generated text to sway human decisions and also points out a new direction for evaluating generated text, i.e., leveraging the reactions and decisions of readers. We release our dataset to assist future research. |
| title | Beyond Turing Test: Can GPT-4 Sway Experts' Decisions? |
| topic | Computational Engineering, Finance, and Science Computation and Language |
| url | https://arxiv.org/abs/2409.16710 |