Adaptive Decoding via Latent Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910698109403136 |
|---|---|
| author | Dhuliawala, Shehzaad Kulikov, Ilia Yu, Ping Celikyilmaz, Asli Weston, Jason Sukhbaatar, Sainbayar Lanchantin, Jack |
| author_facet | Dhuliawala, Shehzaad Kulikov, Ilia Yu, Ping Celikyilmaz, Asli Weston, Jason Sukhbaatar, Sainbayar Lanchantin, Jack |
| contents | During language model decoding, it is known that using higher temperature sampling gives more creative responses, while lower temperatures are more factually accurate. However, such models are commonly applied to general instruction following, which involves both creative and fact seeking tasks, using a single fixed temperature across all examples and tokens. In this work, we introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time, at either the token or example level, in order to optimize performance. To learn its parameters we introduce Latent Preference Optimization (LPO) a general approach to train discrete latent variables such as choices of temperature. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures, including UltraFeedback, Creative Story Writing, and GSM8K. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_09661 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Decoding via Latent Preference Optimization Dhuliawala, Shehzaad Kulikov, Ilia Yu, Ping Celikyilmaz, Asli Weston, Jason Sukhbaatar, Sainbayar Lanchantin, Jack Computation and Language During language model decoding, it is known that using higher temperature sampling gives more creative responses, while lower temperatures are more factually accurate. However, such models are commonly applied to general instruction following, which involves both creative and fact seeking tasks, using a single fixed temperature across all examples and tokens. In this work, we introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time, at either the token or example level, in order to optimize performance. To learn its parameters we introduce Latent Preference Optimization (LPO) a general approach to train discrete latent variables such as choices of temperature. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures, including UltraFeedback, Creative Story Writing, and GSM8K. |
| title | Adaptive Decoding via Latent Preference Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.09661 |