Beemo: Benchmark of Expert-edited Machine-generated Outputs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909539272491008 |
|---|---|
| author | Artemova, Ekaterina Lucas, Jason Venkatraman, Saranya Lee, Jooyoung Tilga, Sergei Uchendu, Adaku Mikhailov, Vladislav |
| author_facet | Artemova, Ekaterina Lucas, Jason Venkatraman, Saranya Lee, Jooyoung Tilga, Sergei Uchendu, Adaku Mikhailov, Vladislav |
| contents | The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04032 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beemo: Benchmark of Expert-edited Machine-generated Outputs Artemova, Ekaterina Lucas, Jason Venkatraman, Saranya Lee, Jooyoung Tilga, Sergei Uchendu, Adaku Mikhailov, Vladislav Computation and Language The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available. |
| title | Beemo: Benchmark of Expert-edited Machine-generated Outputs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.04032 |