Simple LLM Baselines are Competitive for Model Diffing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911439644524544 |
|---|---|
| author | Kempf, Elias Schrodi, Simon Cywiński, Bartosz Brox, Thomas Nanda, Neel Conmy, Arthur |
| author_facet | Kempf, Elias Schrodi, Simon Cywiński, Bartosz Brox, Thomas Nanda, Neel Conmy, Arthur |
| contents | Standard LLM evaluations only test capabilities or dispositions that evaluators designed them for, missing unexpected differences such as behavioral shifts between model revisions or emergent misaligned tendencies. Model diffing addresses this limitation by automatically surfacing systematic behavioral differences. Recent approaches include LLM-based methods that generate natural language descriptions and sparse autoencoder (SAE)-based methods that identify interpretable features. However, no systematic comparison of these approaches exists nor are there established evaluation criteria. We address this gap by proposing evaluation metrics for key desiderata (generalization, interestingness, and abstraction level) and use these to compare existing methods. Our results show that an improved LLM-based baseline performs comparably to the SAE-based method while typically surfacing more abstract behavioral differences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_10371 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Simple LLM Baselines are Competitive for Model Diffing Kempf, Elias Schrodi, Simon Cywiński, Bartosz Brox, Thomas Nanda, Neel Conmy, Arthur Machine Learning Standard LLM evaluations only test capabilities or dispositions that evaluators designed them for, missing unexpected differences such as behavioral shifts between model revisions or emergent misaligned tendencies. Model diffing addresses this limitation by automatically surfacing systematic behavioral differences. Recent approaches include LLM-based methods that generate natural language descriptions and sparse autoencoder (SAE)-based methods that identify interpretable features. However, no systematic comparison of these approaches exists nor are there established evaluation criteria. We address this gap by proposing evaluation metrics for key desiderata (generalization, interestingness, and abstraction level) and use these to compare existing methods. Our results show that an improved LLM-based baseline performs comparably to the SAE-based method while typically surfacing more abstract behavioral differences. |
| title | Simple LLM Baselines are Competitive for Model Diffing |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.10371 |