Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.12639 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911384017567744 |
|---|---|
| author | Vennemeyer, Daniel Pandey, Punya Syon Duong, Phan Anh Umeokoli, Michael Ratnam, Samuel |
| author_facet | Vennemeyer, Daniel Pandey, Punya Syon Duong, Phan Anh Umeokoli, Michael Ratnam, Samuel |
| contents | Fine-tuning LLMs on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes remain limited. We present a controlled comparison of six fine-tuning objectives -- Supervised Fine-Tuning, Direct Preference Optimization, Conditional Fine-Tuning, Inoculation Prompting, Odds Ratio Preference Optimization, and KL-regularized fine-tuning -- holding data, domain, architecture, and optimization fixed. Across closed-form reasoning and open-ended generation tasks, we find that objective choice induces systematic, scale-dependent shifts along the safety-capability frontier. At small training budgets, robustness is similar across objectives but capability differs. At larger budgets, objectives diverge sharply: supervised and preference-based tuning tightly couple capability gains to increased adversarial vulnerability and persona drift, while objectives that constrain learning signals -- especially ORPO and KL-regularization -- substantially mitigate both. Fine-tuning objectives therefore matter little for safety at small scales but become a primary driver of adversarial robustness and latent persona stability as training scale increases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_12639 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift Vennemeyer, Daniel Pandey, Punya Syon Duong, Phan Anh Umeokoli, Michael Ratnam, Samuel Computation and Language Machine Learning Fine-tuning LLMs on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes remain limited. We present a controlled comparison of six fine-tuning objectives -- Supervised Fine-Tuning, Direct Preference Optimization, Conditional Fine-Tuning, Inoculation Prompting, Odds Ratio Preference Optimization, and KL-regularized fine-tuning -- holding data, domain, architecture, and optimization fixed. Across closed-form reasoning and open-ended generation tasks, we find that objective choice induces systematic, scale-dependent shifts along the safety-capability frontier. At small training budgets, robustness is similar across objectives but capability differs. At larger budgets, objectives diverge sharply: supervised and preference-based tuning tightly couple capability gains to increased adversarial vulnerability and persona drift, while objectives that constrain learning signals -- especially ORPO and KL-regularization -- substantially mitigate both. Fine-tuning objectives therefore matter little for safety at small scales but become a primary driver of adversarial robustness and latent persona stability as training scale increases. |
| title | Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.12639 |