Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.20956 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912505799901184 |
|---|---|
| author | Peeperkorn, Max Kouwenhoven, Tom Brown, Dan Jordanous, Anna |
| author_facet | Peeperkorn, Max Kouwenhoven, Tom Brown, Dan Jordanous, Anna |
| contents | Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_20956 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models Peeperkorn, Max Kouwenhoven, Tom Brown, Dan Jordanous, Anna Computation and Language Artificial Intelligence Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality. |
| title | Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.20956 |