Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916659337363456 |
|---|---|
| author | Khan, Arham Nief, Todd Hudson, Nathaniel Sakarvadia, Mansi Grzenda, Daniel Ajith, Aswathy Pettyjohn, Jordan Chard, Kyle Foster, Ian |
| author_facet | Khan, Arham Nief, Todd Hudson, Nathaniel Sakarvadia, Mansi Grzenda, Daniel Ajith, Aswathy Pettyjohn, Jordan Chard, Kyle Foster, Ian |
| contents | We survey the model merging literature through the lens of loss landscape geometry to connect observations from empirical studies on model merging and loss landscape analysis to phenomena that govern neural network training and the emergence of their inner representations. We distill repeated empirical observations from the literature in these fields into descriptions of four major characteristics of loss landscape geometry: mode convexity, determinism, directedness, and connectivity. We argue that insights into the structure of learned representations from model merging have applications to model interpretability and robustness, subsequently we propose promising new research directions at the intersection of these fields. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12927 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey Khan, Arham Nief, Todd Hudson, Nathaniel Sakarvadia, Mansi Grzenda, Daniel Ajith, Aswathy Pettyjohn, Jordan Chard, Kyle Foster, Ian Machine Learning Artificial Intelligence We survey the model merging literature through the lens of loss landscape geometry to connect observations from empirical studies on model merging and loss landscape analysis to phenomena that govern neural network training and the emergence of their inner representations. We distill repeated empirical observations from the literature in these fields into descriptions of four major characteristics of loss landscape geometry: mode convexity, determinism, directedness, and connectivity. We argue that insights into the structure of learned representations from model merging have applications to model interpretability and robustness, subsequently we propose promising new research directions at the intersection of these fields. |
| title | Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.12927 |