Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.00553 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915804221538304 |
|---|---|
| author | Shaib, Chantal Govindarajan, Venkata S. Barrow, Joe Sun, Jiuding Siu, Alexa F. Wallace, Byron C. Nenkova, Ani |
| author_facet | Shaib, Chantal Govindarajan, Venkata S. Barrow, Joe Sun, Jiuding Siu, Alexa F. Wallace, Byron C. Nenkova, Ani |
| contents | The diversity across outputs generated by LLMs shapes perception of their quality and utility. High lexical diversity is often desirable, but there is no standard method to measure this property. Templated answer structures and ``canned'' responses across different documents are readily noticeable, but difficult to visualize across large corpora. This work aims to standardize measurement of text diversity. Specifically, we empirically investigate the convergent validity of existing scores across English texts, and we release diversity, an open-source Python package for measuring and extracting repetition in text. We also build a platform based on diversity for users to interactively explore repetition in text. We find that fast compression algorithms capture information similar to what is measured by slow-to-compute $n$-gram overlap homogeneity scores. Further, a combination of measures -- compression ratios, self-repetition of long $n$-grams, and Self-BLEU and BERTScore -- are sufficient to report, as they have low mutual correlation with each other. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00553 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores Shaib, Chantal Govindarajan, Venkata S. Barrow, Joe Sun, Jiuding Siu, Alexa F. Wallace, Byron C. Nenkova, Ani Computation and Language The diversity across outputs generated by LLMs shapes perception of their quality and utility. High lexical diversity is often desirable, but there is no standard method to measure this property. Templated answer structures and ``canned'' responses across different documents are readily noticeable, but difficult to visualize across large corpora. This work aims to standardize measurement of text diversity. Specifically, we empirically investigate the convergent validity of existing scores across English texts, and we release diversity, an open-source Python package for measuring and extracting repetition in text. We also build a platform based on diversity for users to interactively explore repetition in text. We find that fast compression algorithms capture information similar to what is measured by slow-to-compute $n$-gram overlap homogeneity scores. Further, a combination of measures -- compression ratios, self-repetition of long $n$-grams, and Self-BLEU and BERTScore -- are sufficient to report, as they have low mutual correlation with each other. |
| title | Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.00553 |