Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917377772355584 |
|---|---|
| author | Lee, Gyuseok Kweon, Wonbin Yue, Zhenrui Kang, SeongKu Han, Jiawei Wang, Dong |
| author_facet | Lee, Gyuseok Kweon, Wonbin Yue, Zhenrui Kang, SeongKu Han, Jiawei Wang, Dong |
| contents | Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_00997 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization Lee, Gyuseok Kweon, Wonbin Yue, Zhenrui Kang, SeongKu Han, Jiawei Wang, Dong Computation and Language Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. |
| title | Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.00997 |