Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912794426736640 |
|---|---|
| author | CH-Wang, Sky Svegliato, Justin Appel, Helen Eisner, Jason |
| author_facet | CH-Wang, Sky Svegliato, Justin Appel, Helen Eisner, Jason |
| contents | We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23693 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans CH-Wang, Sky Svegliato, Justin Appel, Helen Eisner, Jason Computation and Language I.2.7 We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning. |
| title | Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2512.23693 |