Scaling Human Judgment in Community Notes with LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911186615795712 |
|---|---|
| author | Li, Haiwen De, Soham Revel, Manon Haupt, Andreas Miller, Brad Coleman, Keith Baxter, Jay Saveski, Martin Bakker, Michiel A. |
| author_facet | Li, Haiwen De, Soham Revel, Manon Haupt, Andreas Miller, Brad Coleman, Keith Baxter, Jay Saveski, Martin Bakker, Michiel A. |
| contents | This paper argues for a new paradigm for Community Notes in the LLM era: an open ecosystem where both humans and LLMs can write notes, and the decision of which notes are helpful enough to show remains in the hands of humans. This approach can accelerate the delivery of notes, while maintaining trust and legitimacy through Community Notes' foundational principle: A community of diverse human raters collectively serve as the ultimate evaluator and arbiter of what is helpful. Further, the feedback from this diverse community can be used to improve LLMs' ability to produce accurate, unbiased, broadly helpful notes--what we term Reinforcement Learning from Community Feedback (RLCF). This becomes a two-way street: LLMs serve as an asset to humans--helping deliver context quickly and with minimal effort--while human feedback, in turn, enhances the performance of LLMs. This paper describes how such a system can work, its benefits, key new risks and challenges it introduces, and a research agenda to solve those challenges and realize the potential of this approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_24118 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Human Judgment in Community Notes with LLMs Li, Haiwen De, Soham Revel, Manon Haupt, Andreas Miller, Brad Coleman, Keith Baxter, Jay Saveski, Martin Bakker, Michiel A. Computers and Society Social and Information Networks This paper argues for a new paradigm for Community Notes in the LLM era: an open ecosystem where both humans and LLMs can write notes, and the decision of which notes are helpful enough to show remains in the hands of humans. This approach can accelerate the delivery of notes, while maintaining trust and legitimacy through Community Notes' foundational principle: A community of diverse human raters collectively serve as the ultimate evaluator and arbiter of what is helpful. Further, the feedback from this diverse community can be used to improve LLMs' ability to produce accurate, unbiased, broadly helpful notes--what we term Reinforcement Learning from Community Feedback (RLCF). This becomes a two-way street: LLMs serve as an asset to humans--helping deliver context quickly and with minimal effort--while human feedback, in turn, enhances the performance of LLMs. This paper describes how such a system can work, its benefits, key new risks and challenges it introduces, and a research agenda to solve those challenges and realize the potential of this approach. |
| title | Scaling Human Judgment in Community Notes with LLMs |
| topic | Computers and Society Social and Information Networks |
| url | https://arxiv.org/abs/2506.24118 |