Decoding-time Realignment of Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911885425639424 |
|---|---|
| author | Liu, Tianlin Guo, Shangmin Bianco, Leonardo Calandriello, Daniele Berthet, Quentin Llinares, Felipe Hoffmann, Jessica Dixon, Lucas Valko, Michal Blondel, Mathieu |
| author_facet | Liu, Tianlin Guo, Shangmin Bianco, Leonardo Calandriello, Daniele Berthet, Quentin Llinares, Felipe Hoffmann, Jessica Dixon, Lucas Valko, Michal Blondel, Mathieu |
| contents | Aligning language models with human preferences is crucial for reducing errors and biases in these models. Alignment techniques, such as reinforcement learning from human feedback (RLHF), are typically cast as optimizing a tradeoff between human preference rewards and a proximity regularization term that encourages staying close to the unaligned model. Selecting an appropriate level of regularization is critical: insufficient regularization can lead to reduced model capabilities due to reward hacking, whereas excessive regularization hinders alignment. Traditional methods for finding the optimal regularization level require retraining multiple models with varying regularization strengths. This process, however, is resource-intensive, especially for large models. To address this challenge, we propose decoding-time realignment (DeRa), a simple method to explore and evaluate different regularization strengths in aligned models without retraining. DeRa enables control over the degree of alignment, allowing users to smoothly transition between unaligned and aligned models. It also enhances the efficiency of hyperparameter tuning by enabling the identification of effective regularization strengths using a validation dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02992 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Decoding-time Realignment of Language Models Liu, Tianlin Guo, Shangmin Bianco, Leonardo Calandriello, Daniele Berthet, Quentin Llinares, Felipe Hoffmann, Jessica Dixon, Lucas Valko, Michal Blondel, Mathieu Machine Learning Artificial Intelligence Computation and Language Aligning language models with human preferences is crucial for reducing errors and biases in these models. Alignment techniques, such as reinforcement learning from human feedback (RLHF), are typically cast as optimizing a tradeoff between human preference rewards and a proximity regularization term that encourages staying close to the unaligned model. Selecting an appropriate level of regularization is critical: insufficient regularization can lead to reduced model capabilities due to reward hacking, whereas excessive regularization hinders alignment. Traditional methods for finding the optimal regularization level require retraining multiple models with varying regularization strengths. This process, however, is resource-intensive, especially for large models. To address this challenge, we propose decoding-time realignment (DeRa), a simple method to explore and evaluate different regularization strengths in aligned models without retraining. DeRa enables control over the degree of alignment, allowing users to smoothly transition between unaligned and aligned models. It also enhances the efficiency of hyperparameter tuning by enabling the identification of effective regularization strengths using a validation dataset. |
| title | Decoding-time Realignment of Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.02992 |