WorldPM: Scaling Human Preference Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908368462938112 |
|---|---|
| author | Wang, Binghai Lin, Runji Lu, Keming Yu, Le Zhang, Zhenru Huang, Fei Zheng, Chujie Dang, Kai Fan, Yang Ren, Xingzhang Yang, An Hui, Binyuan Liu, Dayiheng Gui, Tao Zhang, Qi Huang, Xuanjing Jiang, Yu-Gang Yu, Bowen Zhou, Jingren Lin, Junyang |
| author_facet | Wang, Binghai Lin, Runji Lu, Keming Yu, Le Zhang, Zhenru Huang, Fei Zheng, Chujie Dang, Kai Fan, Yang Ren, Xingzhang Yang, An Hui, Binyuan Liu, Dayiheng Gui, Tao Zhang, Qi Huang, Xuanjing Jiang, Yu-Gang Yu, Bowen Zhou, Jingren Lin, Junyang |
| contents | Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10527 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WorldPM: Scaling Human Preference Modeling Wang, Binghai Lin, Runji Lu, Keming Yu, Le Zhang, Zhenru Huang, Fei Zheng, Chujie Dang, Kai Fan, Yang Ren, Xingzhang Yang, An Hui, Binyuan Liu, Dayiheng Gui, Tao Zhang, Qi Huang, Xuanjing Jiang, Yu-Gang Yu, Bowen Zhou, Jingren Lin, Junyang Computation and Language Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations. |
| title | WorldPM: Scaling Human Preference Modeling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.10527 |