BOND: Aligning LLMs with Best-of-N Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909262917140480 |
|---|---|
| author | Sessa, Pier Giuseppe Dadashi, Robert Hussenot, Léonard Ferret, Johan Vieillard, Nino Ramé, Alexandre Shariari, Bobak Perrin, Sarah Friesen, Abe Cideron, Geoffrey Girgin, Sertan Stanczyk, Piotr Michi, Andrea Sinopalnikov, Danila Ramos, Sabela Héliou, Amélie Severyn, Aliaksei Hoffman, Matt Momchev, Nikola Bachem, Olivier |
| author_facet | Sessa, Pier Giuseppe Dadashi, Robert Hussenot, Léonard Ferret, Johan Vieillard, Nino Ramé, Alexandre Shariari, Bobak Perrin, Sarah Friesen, Abe Cideron, Geoffrey Girgin, Sertan Stanczyk, Piotr Michi, Andrea Sinopalnikov, Danila Ramos, Sabela Héliou, Amélie Severyn, Aliaksei Hoffman, Matt Momchev, Nikola Bachem, Olivier |
| contents | Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models. Aligning Gemma policies with BOND outperforms other RLHF algorithms by improving results on several benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14622 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BOND: Aligning LLMs with Best-of-N Distillation Sessa, Pier Giuseppe Dadashi, Robert Hussenot, Léonard Ferret, Johan Vieillard, Nino Ramé, Alexandre Shariari, Bobak Perrin, Sarah Friesen, Abe Cideron, Geoffrey Girgin, Sertan Stanczyk, Piotr Michi, Andrea Sinopalnikov, Danila Ramos, Sabela Héliou, Amélie Severyn, Aliaksei Hoffman, Matt Momchev, Nikola Bachem, Olivier Machine Learning Artificial Intelligence Computation and Language Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models. Aligning Gemma policies with BOND outperforms other RLHF algorithms by improving results on several benchmarks. |
| title | BOND: Aligning LLMs with Best-of-N Distillation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.14622 |