Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.02646 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908299434131456 |
|---|---|
| author | Kiyohara, Haruka Cao, Daniel Yiming Saito, Yuta Joachims, Thorsten |
| author_facet | Kiyohara, Haruka Cao, Daniel Yiming Saito, Yuta Joachims, Thorsten |
| contents | We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02646 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt Optimization with Logged Bandit Data Kiyohara, Haruka Cao, Daniel Yiming Saito, Yuta Joachims, Thorsten Machine Learning Artificial Intelligence Information Retrieval We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large. |
| title | Prompt Optimization with Logged Bandit Data |
| topic | Machine Learning Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2504.02646 |