Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909663972294656 |
|---|---|
| author | Xiao, Yao Ye, Hai Chen, Linyao Ng, Hwee Tou Bing, Lidong Li, Xiaoli Lee, Roy Ka-wei |
| author_facet | Xiao, Yao Ye, Hai Chen, Linyao Ng, Hwee Tou Bing, Lidong Li, Xiaoli Lee, Roy Ka-wei |
| contents | Iterative data generation and model retraining are widely used to align large language models (LLMs). It typically involves a policy model to generate on-policy responses and a reward model to guide training data selection. Direct Preference Optimization (DPO) further enhances this process by constructing preference pairs of chosen and rejected responses. In this work, we aim to \emph{scale up} the number of on-policy samples via repeated random sampling to improve alignment performance. Conventional practice selects the sample with the highest reward as chosen and the lowest as rejected for DPO. However, our experiments reveal that this strategy leads to a \emph{decline} in performance as the sample size increases. To address this, we investigate preference data construction through the lens of underlying normal distribution of sample rewards. We categorize the reward space into seven representative points and systematically explore all 21 ($C_7^2$) pairwise combinations. Through evaluations on four models using AlpacaEval 2, we find that selecting the rejected response at reward position $μ- 2σ$ rather than the minimum reward, is crucial for optimal performance. We finally introduce a scalable preference data construction strategy that consistently enhances model performance as the sample scale increases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16825 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization Xiao, Yao Ye, Hai Chen, Linyao Ng, Hwee Tou Bing, Lidong Li, Xiaoli Lee, Roy Ka-wei Computation and Language Iterative data generation and model retraining are widely used to align large language models (LLMs). It typically involves a policy model to generate on-policy responses and a reward model to guide training data selection. Direct Preference Optimization (DPO) further enhances this process by constructing preference pairs of chosen and rejected responses. In this work, we aim to \emph{scale up} the number of on-policy samples via repeated random sampling to improve alignment performance. Conventional practice selects the sample with the highest reward as chosen and the lowest as rejected for DPO. However, our experiments reveal that this strategy leads to a \emph{decline} in performance as the sample size increases. To address this, we investigate preference data construction through the lens of underlying normal distribution of sample rewards. We categorize the reward space into seven representative points and systematically explore all 21 ($C_7^2$) pairwise combinations. Through evaluations on four models using AlpacaEval 2, we find that selecting the rejected response at reward position $μ- 2σ$ rather than the minimum reward, is crucial for optimal performance. We finally introduce a scalable preference data construction strategy that consistently enhances model performance as the sample scale increases. |
| title | Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.16825 |