Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.11344 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909690060865536 |
|---|---|
| author | Hall, Zara Subbiah, Melanie Zollo, Thomas P McKeown, Kathleen Zemel, Richard |
| author_facet | Hall, Zara Subbiah, Melanie Zollo, Thomas P McKeown, Kathleen Zemel, Richard |
| contents | Large language models are increasingly used to support high-stakes decisions, potentially influencing who is granted bail or receives a loan. Naive chain-of-thought sampling can improve average decision accuracy, but has also been shown to amplify unfair bias. To address this challenge and enable the trustworthy use of reasoning models in high-stakes decision-making, we propose a framework for training a generalizable Fairness Reward Model (FRM). Our model assigns a fairness score to LLM reasoning, enabling the system to down-weight biased trajectories and favor equitable ones when aggregating decisions across reasoning chains. We show that a single Fairness Reward Model, trained on weakly supervised, LLM-annotated examples of biased versus unbiased reasoning, transfers across tasks, domains, and model families without additional fine-tuning. Applied to real-world decision-making tasks including recidivism prediction and social media moderation, we show that our approach consistently improves fairness while matching, or even surpassing, baseline accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_11344 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Guiding LLM Decision-Making with Fairness Reward Models Hall, Zara Subbiah, Melanie Zollo, Thomas P McKeown, Kathleen Zemel, Richard Machine Learning Large language models are increasingly used to support high-stakes decisions, potentially influencing who is granted bail or receives a loan. Naive chain-of-thought sampling can improve average decision accuracy, but has also been shown to amplify unfair bias. To address this challenge and enable the trustworthy use of reasoning models in high-stakes decision-making, we propose a framework for training a generalizable Fairness Reward Model (FRM). Our model assigns a fairness score to LLM reasoning, enabling the system to down-weight biased trajectories and favor equitable ones when aggregating decisions across reasoning chains. We show that a single Fairness Reward Model, trained on weakly supervised, LLM-annotated examples of biased versus unbiased reasoning, transfers across tasks, domains, and model families without additional fine-tuning. Applied to real-world decision-making tasks including recidivism prediction and social media moderation, we show that our approach consistently improves fairness while matching, or even surpassing, baseline accuracy. |
| title | Guiding LLM Decision-Making with Fairness Reward Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.11344 |