MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909033373368320 |
|---|---|
| author | Surana, Rohan Li, Xintong Yu, Sheldon Shen, Yiran Jenny Wang, Chuhan Yu, Tong Ammanabrolu, Prithviraj Shang, Jingbo McAuley, Julian Wu, Junda |
| author_facet | Surana, Rohan Li, Xintong Yu, Sheldon Shen, Yiran Jenny Wang, Chuhan Yu, Tong Ammanabrolu, Prithviraj Shang, Jingbo McAuley, Julian Wu, Junda |
| contents | Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10784 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization Surana, Rohan Li, Xintong Yu, Sheldon Shen, Yiran Jenny Wang, Chuhan Yu, Tong Ammanabrolu, Prithviraj Shang, Jingbo McAuley, Julian Wu, Junda Machine Learning Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives. |
| title | MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.10784 |