Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912911105982464 |
|---|---|
| author | Prasad, Aaditya Vikram Watts, Connor Merullo, Jack Gala, Dhruvil Lewis, Owen McGrath, Thomas Lubana, Ekdeep Singh |
| author_facet | Prasad, Aaditya Vikram Watts, Connor Merullo, Jack Gala, Dhruvil Lewis, Owen McGrath, Thomas Lubana, Ekdeep Singh |
| contents | Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used for test-time monitoring or steering. We present an alternative affordance: features as scalable supervision for open-ended tasks. We consider the case of hallucination-reduction as a desirable, yet open-ended behavior and design a reinforcement learning (RL) pipeline, titled RLFR (Reinforcement Learning from Feature Rewards), that uses features as reward functions. Grounded in a novel probing framework that identifies candidate hallucinated claims, our pipeline teaches a model to intervene and correct its completions when it is uncertain of their factuality. Furthermore, the pipeline enables scalable test-time compute, guided once more by our reward features. This end-to-end process operationalized on Gemma-3-12B-IT results in a policy that is 58% less likely to hallucinate compared to the original model (when run in tandem with our probing harness), while preserving performance on standard benchmarks. Taken together, by grounding supervision in the language of features, this paper introduces a novel paradigm in the use of interpretability for learning open-ended tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_10067 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability Prasad, Aaditya Vikram Watts, Connor Merullo, Jack Gala, Dhruvil Lewis, Owen McGrath, Thomas Lubana, Ekdeep Singh Machine Learning Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used for test-time monitoring or steering. We present an alternative affordance: features as scalable supervision for open-ended tasks. We consider the case of hallucination-reduction as a desirable, yet open-ended behavior and design a reinforcement learning (RL) pipeline, titled RLFR (Reinforcement Learning from Feature Rewards), that uses features as reward functions. Grounded in a novel probing framework that identifies candidate hallucinated claims, our pipeline teaches a model to intervene and correct its completions when it is uncertain of their factuality. Furthermore, the pipeline enables scalable test-time compute, guided once more by our reward features. This end-to-end process operationalized on Gemma-3-12B-IT results in a policy that is 58% less likely to hallucinate compared to the original model (when run in tandem with our probing harness), while preserving performance on standard benchmarks. Taken together, by grounding supervision in the language of features, this paper introduces a novel paradigm in the use of interpretability for learning open-ended tasks. |
| title | Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.10067 |