Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2605.09995 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918494007721984 |
|---|---|
| author | Springer, Jacob Mitchell Advani, Madhu Aichberger, Lukas Bradley, Arwen Malach, Eran Saremi, Omid Williamson, Sinead Nakkiran, Preetum Littwin, Etai Raghunathan, Aditi |
| author_facet | Springer, Jacob Mitchell Advani, Madhu Aichberger, Lukas Bradley, Arwen Malach, Eran Saremi, Omid Williamson, Sinead Nakkiran, Preetum Littwin, Etai Raghunathan, Aditi |
| contents | Post-training (via supervised fine-tuning) improves instruction-following, but often induces semantic mode collapse by biasing models toward low-entropy fine-tuning data at the expense of the high-entropy pretraining distribution. Crucially, we find this trade-off worsens with scale. To close this semantic diversity gap, we propose annotation-anchored training, a principled method that enables models to adopt the preference-following behaviors of post-training without sacrificing the inherent diversity of pretraining. Our approach is simple: we pretrain on documents paired with semantic annotations, inducing a rich annotation distribution that reflects the full breadth of pretraining data, and we preserve this distribution during post-training. This lets us sample diverse annotations at inference time and use them as anchors to guide generation, effectively transferring pretraining's semantic richness into post-trained models. We find that models trained with annotation-anchored training can attain $6 \times$ less diversity collapse than models trained with SFT, and improve with scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09995 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Annotations Mitigate Post-Training Mode Collapse Springer, Jacob Mitchell Advani, Madhu Aichberger, Lukas Bradley, Arwen Malach, Eran Saremi, Omid Williamson, Sinead Nakkiran, Preetum Littwin, Etai Raghunathan, Aditi Computation and Language Post-training (via supervised fine-tuning) improves instruction-following, but often induces semantic mode collapse by biasing models toward low-entropy fine-tuning data at the expense of the high-entropy pretraining distribution. Crucially, we find this trade-off worsens with scale. To close this semantic diversity gap, we propose annotation-anchored training, a principled method that enables models to adopt the preference-following behaviors of post-training without sacrificing the inherent diversity of pretraining. Our approach is simple: we pretrain on documents paired with semantic annotations, inducing a rich annotation distribution that reflects the full breadth of pretraining data, and we preserve this distribution during post-training. This lets us sample diverse annotations at inference time and use them as anchors to guide generation, effectively transferring pretraining's semantic richness into post-trained models. We find that models trained with annotation-anchored training can attain $6 \times$ less diversity collapse than models trained with SFT, and improve with scale. |
| title | Annotations Mitigate Post-Training Mode Collapse |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.09995 |