TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917985469333504 |
|---|---|
| author | Liu, Aiwei Bai, Haoping Lu, Zhiyun Sun, Yanchao Kong, Xiang Wang, Simon Shan, Jiulong Jose, Albin Madappally Liu, Xiaojiang Wen, Lijie Yu, Philip S. Cao, Meng |
| author_facet | Liu, Aiwei Bai, Haoping Lu, Zhiyun Sun, Yanchao Kong, Xiang Wang, Simon Shan, Jiulong Jose, Albin Madappally Liu, Xiaojiang Wen, Lijie Yu, Philip S. Cao, Meng |
| contents | Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results. In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses. Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04350 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights Liu, Aiwei Bai, Haoping Lu, Zhiyun Sun, Yanchao Kong, Xiang Wang, Simon Shan, Jiulong Jose, Albin Madappally Liu, Xiaojiang Wen, Lijie Yu, Philip S. Cao, Meng Computation and Language 68T50 I.2.7 Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results. In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses. Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions. |
| title | TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights |
| topic | Computation and Language 68T50 I.2.7 |
| url | https://arxiv.org/abs/2410.04350 |