SteerEval: A Framework for Evaluating Steerability with Natural Language Profiles for Recommendation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917230537605120 |
|---|---|
| author | Zhou, Joyce Zhou, Weijie Turnbull, Doug Joachims, Thorsten |
| author_facet | Zhou, Joyce Zhou, Weijie Turnbull, Doug Joachims, Thorsten |
| contents | Natural-language user profiles have recently attracted attention not only for improved interpretability, but also for their potential to make recommender systems more steerable. By enabling direct editing, natural-language profiles allow users to explicitly articulate preferences that may be difficult to infer from past behavior. However, it remains unclear whether current natural-language-based recommendation methods can follow such steering commands. While existing steerability evaluations have shown some success for well-recognized item attributes (e.g., movie genres), we argue that these benchmarks fail to capture the richer forms of user control that motivate steerable recommendations. To address this gap, we introduce SteerEval, an evaluation framework designed to measure more nuanced and diverse forms of steerability by using interventions that range from genres to content-warning for movies. We assess the steerability of a family of pretrained natural-language recommenders, examine the potential and limitations of steering on relatively niche topics, and compare how different profile and recommendation interventions impact steering effectiveness. Finally, we offer practical design suggestions informed by our findings and discuss future steps in steerable recommender design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21105 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SteerEval: A Framework for Evaluating Steerability with Natural Language Profiles for Recommendation Zhou, Joyce Zhou, Weijie Turnbull, Doug Joachims, Thorsten Information Retrieval Artificial Intelligence Computation and Language Human-Computer Interaction Natural-language user profiles have recently attracted attention not only for improved interpretability, but also for their potential to make recommender systems more steerable. By enabling direct editing, natural-language profiles allow users to explicitly articulate preferences that may be difficult to infer from past behavior. However, it remains unclear whether current natural-language-based recommendation methods can follow such steering commands. While existing steerability evaluations have shown some success for well-recognized item attributes (e.g., movie genres), we argue that these benchmarks fail to capture the richer forms of user control that motivate steerable recommendations. To address this gap, we introduce SteerEval, an evaluation framework designed to measure more nuanced and diverse forms of steerability by using interventions that range from genres to content-warning for movies. We assess the steerability of a family of pretrained natural-language recommenders, examine the potential and limitations of steering on relatively niche topics, and compare how different profile and recommendation interventions impact steering effectiveness. Finally, we offer practical design suggestions informed by our findings and discuss future steps in steerable recommender design. |
| title | SteerEval: A Framework for Evaluating Steerability with Natural Language Profiles for Recommendation |
| topic | Information Retrieval Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2601.21105 |