Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.00680 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916932916084736 |
|---|---|
| author | Römisch, Johannes Gorovaia, Svetlana Halchynska, Mariia Schmidt, Gleb Yamshchikov, Ivan P. |
| author_facet | Römisch, Johannes Gorovaia, Svetlana Halchynska, Mariia Schmidt, Gleb Yamshchikov, Ivan P. |
| contents | This article explores the zero-shot performance of state-of-the-art large language models (LLMs) on one of the most challenging tasks in authorship analysis: sentence-level style change detection. Benchmarking four LLMs on the official PAN~2024 and 2025 "Multi-Author Writing Style Analysis" datasets, we present several observations. First, state-of-the-art generative models are sensitive to variations in writing style - even at the granular level of individual sentences. Second, their accuracy establishes a challenging baseline for the task, outperforming suggested baselines of the PAN competition. Finally, we explore the influence of semantics on model predictions and present evidence suggesting that the latest generation of LLMs may be more sensitive to content-independent and purely stylistic signals than previously reported. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00680 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Better Call Claude: Can LLMs Detect Changes of Writing Style? Römisch, Johannes Gorovaia, Svetlana Halchynska, Mariia Schmidt, Gleb Yamshchikov, Ivan P. Computation and Language This article explores the zero-shot performance of state-of-the-art large language models (LLMs) on one of the most challenging tasks in authorship analysis: sentence-level style change detection. Benchmarking four LLMs on the official PAN~2024 and 2025 "Multi-Author Writing Style Analysis" datasets, we present several observations. First, state-of-the-art generative models are sensitive to variations in writing style - even at the granular level of individual sentences. Second, their accuracy establishes a challenging baseline for the task, outperforming suggested baselines of the PAN competition. Finally, we explore the influence of semantics on model predictions and present evidence suggesting that the latest generation of LLMs may be more sensitive to content-independent and purely stylistic signals than previously reported. |
| title | Better Call Claude: Can LLMs Detect Changes of Writing Style? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.00680 |