Exploring Representation Invariance in Finetuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914073988300800 |
|---|---|
| author | Zu, Wenqiang Xie, Shenghao Chen, Hao Chen, Zhiqiang Hu, Liwen Xi, Yuanhao Liang, Yiming Ye, Junliang Lei, Bo Huang, Tiejun Li, Guoqi Ma, Lei |
| author_facet | Zu, Wenqiang Xie, Shenghao Chen, Hao Chen, Zhiqiang Hu, Liwen Xi, Yuanhao Liang, Yiming Ye, Junliang Lei, Bo Huang, Tiejun Li, Guoqi Ma, Lei |
| contents | Foundation models pretrained on large-scale natural images are widely adapted to various cross-domain low-resource downstream tasks, benefiting from generalizable and transferable patterns captured by their representations. However, these representations are later found to gradually vanish during finetuning, accompanied by a degradation of model's original generalizability. In this paper, we argue that such tasks can be effectively adapted without sacrificing the benefits of pretrained representations. We approach this by introducing \textit{Representation Invariance FineTuning (RIFT)}, a regularization that maximizes the representation similarity between pretrained and finetuned models by leveraging orthogonal invariance of manifolds in a computationally efficient way. Experiments demonstrate that our method is compatible with mainstream finetuning methods, offering competitive or even enhanced performance and better preservation of the generalizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07399 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring Representation Invariance in Finetuning Zu, Wenqiang Xie, Shenghao Chen, Hao Chen, Zhiqiang Hu, Liwen Xi, Yuanhao Liang, Yiming Ye, Junliang Lei, Bo Huang, Tiejun Li, Guoqi Ma, Lei Computer Vision and Pattern Recognition Foundation models pretrained on large-scale natural images are widely adapted to various cross-domain low-resource downstream tasks, benefiting from generalizable and transferable patterns captured by their representations. However, these representations are later found to gradually vanish during finetuning, accompanied by a degradation of model's original generalizability. In this paper, we argue that such tasks can be effectively adapted without sacrificing the benefits of pretrained representations. We approach this by introducing \textit{Representation Invariance FineTuning (RIFT)}, a regularization that maximizes the representation similarity between pretrained and finetuned models by leveraging orthogonal invariance of manifolds in a computationally efficient way. Experiments demonstrate that our method is compatible with mainstream finetuning methods, offering competitive or even enhanced performance and better preservation of the generalizability. |
| title | Exploring Representation Invariance in Finetuning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.07399 |