Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909356050612224 |
|---|---|
| author | Arias, Esteban Garces Rodemann, Julian Li, Meimingwei Heumann, Christian Aßenmacher, Matthias |
| author_facet | Arias, Esteban Garces Rodemann, Julian Li, Meimingwei Heumann, Christian Aßenmacher, Matthias |
| contents | Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, nucleus $p-$sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18698 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation Arias, Esteban Garces Rodemann, Julian Li, Meimingwei Heumann, Christian Aßenmacher, Matthias Computation and Language Machine Learning Methodology Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, nucleus $p-$sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available. |
| title | Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation |
| topic | Computation and Language Machine Learning Methodology |
| url | https://arxiv.org/abs/2407.18698 |