Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2407.15762 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909359502524416 |
|---|---|
| author | Wang, Kaiwen Kidambi, Rahul Sullivan, Ryan Agarwal, Alekh Dann, Christoph Michi, Andrea Gelmi, Marco Li, Yunxuan Gupta, Raghav Dubey, Avinava Ramé, Alexandre Ferret, Johan Cideron, Geoffrey Hou, Le Yu, Hongkun Ahmed, Amr Mehta, Aranyak Hussenot, Léonard Bachem, Olivier Leurent, Edouard |
| author_facet | Wang, Kaiwen Kidambi, Rahul Sullivan, Ryan Agarwal, Alekh Dann, Christoph Michi, Andrea Gelmi, Marco Li, Yunxuan Gupta, Raghav Dubey, Avinava Ramé, Alexandre Ferret, Johan Cideron, Geoffrey Hou, Le Yu, Hongkun Ahmed, Amr Mehta, Aranyak Hussenot, Léonard Bachem, Olivier Leurent, Edouard |
| contents | Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15762 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning Wang, Kaiwen Kidambi, Rahul Sullivan, Ryan Agarwal, Alekh Dann, Christoph Michi, Andrea Gelmi, Marco Li, Yunxuan Gupta, Raghav Dubey, Avinava Ramé, Alexandre Ferret, Johan Cideron, Geoffrey Hou, Le Yu, Hongkun Ahmed, Amr Mehta, Aranyak Hussenot, Léonard Bachem, Olivier Leurent, Edouard Machine Learning Artificial Intelligence Computation and Language Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning. |
| title | Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.15762 |