Crowd-SFT: Crowdsourcing for LLM Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915325149184000 |
|---|---|
| author | Sotiropoulos, Alex Valapu, Sulyab Thottungal Lei, Linus Coleman, Jared Krishnamachari, Bhaskar |
| author_facet | Sotiropoulos, Alex Valapu, Sulyab Thottungal Lei, Linus Coleman, Jared Krishnamachari, Bhaskar |
| contents | Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align model responses with human preferences. While RLHF employs a reinforcement learning approach with a separate reward model, SFT uses human-curated datasets for supervised learning. Both approaches traditionally depend on small, vetted groups of annotators, making them costly, prone to bias, and limited in scalability. We propose an open, crowd-sourced fine-tuning framework that addresses these limitations by enabling broader feedback collection for SFT without extensive annotator training. Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates. Our multi-model selection framework demonstrates up to a 55% reduction in target distance over single-model selection, enabling subsequent experiments that validate our point-based reward mechanism's close alignment with Shapley values (a well-established method for attributing individual contributions) thereby supporting fair and scalable participation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04063 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Crowd-SFT: Crowdsourcing for LLM Alignment Sotiropoulos, Alex Valapu, Sulyab Thottungal Lei, Linus Coleman, Jared Krishnamachari, Bhaskar Human-Computer Interaction Distributed, Parallel, and Cluster Computing Machine Learning Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align model responses with human preferences. While RLHF employs a reinforcement learning approach with a separate reward model, SFT uses human-curated datasets for supervised learning. Both approaches traditionally depend on small, vetted groups of annotators, making them costly, prone to bias, and limited in scalability. We propose an open, crowd-sourced fine-tuning framework that addresses these limitations by enabling broader feedback collection for SFT without extensive annotator training. Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates. Our multi-model selection framework demonstrates up to a 55% reduction in target distance over single-model selection, enabling subsequent experiments that validate our point-based reward mechanism's close alignment with Shapley values (a well-established method for attributing individual contributions) thereby supporting fair and scalable participation. |
| title | Crowd-SFT: Crowdsourcing for LLM Alignment |
| topic | Human-Computer Interaction Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2506.04063 |