T-POP: Test-Time Personalization with Online Preference Feedback
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910274464776192 |
|---|---|
| author | Qu, Zikun Zhang, Min Kong, Mingze Li, Xiang Shang, Zhiwei Wang, Zhiyong Ban, Yikun Qiu, Shuang Shu, Yao Dai, Zhongxiang |
| author_facet | Qu, Zikun Zhang, Min Kong, Mingze Li, Xiang Shang, Zhiwei Wang, Zhiyong Ban, Yikun Qiu, Shuang Shu, Yao Dai, Zhongxiang |
| contents | Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback}), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24696 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | T-POP: Test-Time Personalization with Online Preference Feedback Qu, Zikun Zhang, Min Kong, Mingze Li, Xiang Shang, Zhiwei Wang, Zhiyong Ban, Yikun Qiu, Shuang Shu, Yao Dai, Zhongxiang Machine Learning Artificial Intelligence Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses. However, current personalization methods are ill-suited for new users, as they typically require either slow, resource-intensive fine-tuning or a substantial amount of pre-existing user data, creating a significant cold-start problem. To address this challenge, we introduce a new paradigm for real-time personalization by learning from online pairwise preference feedback collected during text generation. We propose T-POP (Test-Time Personalization with Online Preference Feedback}), a novel algorithm that synergistically combines test-time alignment with dueling bandits. Without updating the LLM parameters, T-POP steers the decoding process of a frozen LLM by learning a reward function online that captures user preferences. By leveraging dueling bandits, T-POP intelligently queries the user to efficiently balance between exploring their preferences and exploiting the learned knowledge to generate personalized text. Extensive experiments demonstrate that T-POP achieves rapid and data-efficient personalization, significantly outperforming existing baselines and showing consistent improvement with more user interactions. |
| title | T-POP: Test-Time Personalization with Online Preference Feedback |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.24696 |