RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916950187180032 |
|---|---|
| author | Rupprecht, Timothy Nan, Enfu Akbari, Arash Akbari, Arman Lu, Lei Maan, Priyanka Duffy, Sean Zhao, Pu He, Yumei Kaeli, David Wang, Yanzhi |
| author_facet | Rupprecht, Timothy Nan, Enfu Akbari, Arash Akbari, Arman Lu, Lei Maan, Priyanka Duffy, Sean Zhao, Pu He, Yumei Kaeli, David Wang, Yanzhi |
| contents | Role-playing Large language models (LLMs) are increasingly deployed in high-stakes domains such as healthcare, education, and governance, where failures can directly impact user trust and well-being. A cost effective paradigm for LLM role-playing is few-shot learning, but existing approaches often cause models to break character in unexpected and potentially harmful ways, especially when interacting with hostile users. Inspired by Retrieval-Augmented Generation (RAG), we reformulate LLM role-playing into a text retrieval problem and propose a new prompting framework called RAGs-to-Riches, which leverages curated reference demonstrations to condition LLM responses. We evaluate our framework with LLM-as-a-judge preference voting and introduce two novel token-level ROUGE metrics: Intersection over Output (IOO) to quantity how much an LLM improvises and Intersection over References (IOR) to measure few-shot demonstrations utilization rate during the evaluation tasks. When simulating interactions with a hostile user, our prompting strategy incorporates in its responses during inference an average of 35% more tokens from the reference demonstrations. As a result, across 453 role-playing interactions, our models are consistently judged as being more authentic, and remain in-character more often than zero-shot and in-context Learning (ICL) methods. Our method presents a scalable strategy for building robust, human-aligned LLM role-playing frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12168 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playing Rupprecht, Timothy Nan, Enfu Akbari, Arash Akbari, Arman Lu, Lei Maan, Priyanka Duffy, Sean Zhao, Pu He, Yumei Kaeli, David Wang, Yanzhi Computation and Language Artificial Intelligence Role-playing Large language models (LLMs) are increasingly deployed in high-stakes domains such as healthcare, education, and governance, where failures can directly impact user trust and well-being. A cost effective paradigm for LLM role-playing is few-shot learning, but existing approaches often cause models to break character in unexpected and potentially harmful ways, especially when interacting with hostile users. Inspired by Retrieval-Augmented Generation (RAG), we reformulate LLM role-playing into a text retrieval problem and propose a new prompting framework called RAGs-to-Riches, which leverages curated reference demonstrations to condition LLM responses. We evaluate our framework with LLM-as-a-judge preference voting and introduce two novel token-level ROUGE metrics: Intersection over Output (IOO) to quantity how much an LLM improvises and Intersection over References (IOR) to measure few-shot demonstrations utilization rate during the evaluation tasks. When simulating interactions with a hostile user, our prompting strategy incorporates in its responses during inference an average of 35% more tokens from the reference demonstrations. As a result, across 453 role-playing interactions, our models are consistently judged as being more authentic, and remain in-character more often than zero-shot and in-context Learning (ICL) methods. Our method presents a scalable strategy for building robust, human-aligned LLM role-playing frameworks. |
| title | RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playing |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12168 |