SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915611837202432 |
|---|---|
| author | Liu, Dengcan Li, Jiahao Fu, Zheren Tu, Yi Li, Jiajun Mao, Zhendong Zhang, Yongdong |
| author_facet | Liu, Dengcan Li, Jiahao Fu, Zheren Tu, Yi Li, Jiajun Mao, Zhendong Zhang, Yongdong |
| contents | Reward models (RMs) are a core component in the post-training of large language models (LLMs), serving as proxies for human preference evaluation and guiding model alignment. However, training reliable RMs under limited resources remains challenging due to the reliance on large-scale preference annotations and the high cost of fine-tuning LLMs. To address this, we propose SparseRM, which leverages Sparse Autoencoder (SAE) to extract preference-relevant information encoded in model representations, enabling the construction of a lightweight and interpretable reward model. SparseRM first employs SAE to decompose LLM representations into interpretable directions that capture preference-relevant features. The representations are then projected onto these directions to compute alignment scores, which quantify the strength of each preference feature in the representations. A simple reward head aggregates these scores to predict preference scores. Experiments on three preference modeling tasks show that SparseRM achieves superior performance over most mainstream RMs while using less than 1% of trainable parameters. Moreover, it integrates seamlessly into downstream alignment pipelines, highlighting its potential for efficient alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07896 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder Liu, Dengcan Li, Jiahao Fu, Zheren Tu, Yi Li, Jiajun Mao, Zhendong Zhang, Yongdong Artificial Intelligence Computation and Language Reward models (RMs) are a core component in the post-training of large language models (LLMs), serving as proxies for human preference evaluation and guiding model alignment. However, training reliable RMs under limited resources remains challenging due to the reliance on large-scale preference annotations and the high cost of fine-tuning LLMs. To address this, we propose SparseRM, which leverages Sparse Autoencoder (SAE) to extract preference-relevant information encoded in model representations, enabling the construction of a lightweight and interpretable reward model. SparseRM first employs SAE to decompose LLM representations into interpretable directions that capture preference-relevant features. The representations are then projected onto these directions to compute alignment scores, which quantify the strength of each preference feature in the representations. A simple reward head aggregates these scores to predict preference scores. Experiments on three preference modeling tasks show that SparseRM achieves superior performance over most mainstream RMs while using less than 1% of trainable parameters. Moreover, it integrates seamlessly into downstream alignment pipelines, highlighting its potential for efficient alignment. |
| title | SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2511.07896 |