Interpretable Reward Modeling with Active Concept Bottlenecks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912492469354496 |
|---|---|
| author | Laguna, Sonia Kobalczyk, Katarzyna Vogt, Julia E. Van der Schaar, Mihaela |
| author_facet | Laguna, Sonia Kobalczyk, Katarzyna Vogt, Julia E. Van der Schaar, Mihaela |
| contents | We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04695 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Interpretable Reward Modeling with Active Concept Bottlenecks Laguna, Sonia Kobalczyk, Katarzyna Vogt, Julia E. Van der Schaar, Mihaela Machine Learning We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models. |
| title | Interpretable Reward Modeling with Active Concept Bottlenecks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.04695 |