RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918154620370944 |
|---|---|
| author | Mordo, Tommy Dekel, Sagie Madmon, Omer Tennenholtz, Moshe Kurland, Oren |
| author_facet | Mordo, Tommy Dekel, Sagie Madmon, Omer Tennenholtz, Moshe Kurland, Oren |
| contents | Competitive search is a setting where document publishers modify them to improve their ranking in response to a query. Recently, publishers have increasingly leveraged LLMs to generate and modify competitive content. We introduce Reinforcement Learning from Ranker Feedback (RLRF), a framework that trains LLMs using preference datasets derived from ranking competitions. The goal of a publisher (LLM-based) agent is to optimize content for improved ranking while accounting for the strategies of competing agents. We generate the datasets using approaches that do not rely on human-authored data. We show that our proposed agents consistently and substantially outperform previously suggested approaches for LLM-based competitive document modification. We further show that our agents are effective with ranking functions they were not trained for (i.e., out of distribution) and they adapt to strategic opponents. These findings provide support to the significant potential of using reinforcement learning in competitive search. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04096 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback Mordo, Tommy Dekel, Sagie Madmon, Omer Tennenholtz, Moshe Kurland, Oren Information Retrieval Computer Science and Game Theory Machine Learning Competitive search is a setting where document publishers modify them to improve their ranking in response to a query. Recently, publishers have increasingly leveraged LLMs to generate and modify competitive content. We introduce Reinforcement Learning from Ranker Feedback (RLRF), a framework that trains LLMs using preference datasets derived from ranking competitions. The goal of a publisher (LLM-based) agent is to optimize content for improved ranking while accounting for the strategies of competing agents. We generate the datasets using approaches that do not rely on human-authored data. We show that our proposed agents consistently and substantially outperform previously suggested approaches for LLM-based competitive document modification. We further show that our agents are effective with ranking functions they were not trained for (i.e., out of distribution) and they adapt to strategic opponents. These findings provide support to the significant potential of using reinforcement learning in competitive search. |
| title | RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback |
| topic | Information Retrieval Computer Science and Game Theory Machine Learning |
| url | https://arxiv.org/abs/2510.04096 |