Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911825323360256 |
|---|---|
| author | Liu, Yinhong Su, Yixuan Shareghi, Ehsan Collier, Nigel |
| author_facet | Liu, Yinhong Su, Yixuan Shareghi, Ehsan Collier, Nigel |
| contents | Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective. However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence. The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles. Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10175 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence Liu, Yinhong Su, Yixuan Shareghi, Ehsan Collier, Nigel Computation and Language Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective. However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence. The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles. Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods. |
| title | Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.10175 |