Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916583020953600 |
|---|---|
| author | Michel, Gaspard Epure, Elena V. Hennequin, Romain Cerisara, Christophe |
| author_facet | Michel, Gaspard Epure, Elena V. Hennequin, Romain Cerisara, Christophe |
| contents | Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination. We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11380 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3 Michel, Gaspard Epure, Elena V. Hennequin, Romain Cerisara, Christophe Computation and Language Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination. We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data. |
| title | Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3 |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.11380 |