Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12413 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915297300054016 |
|---|---|
| author | Wang, Jiaqi Zhou, Yuhang Zhang, Zhixiong Chen, Qiguang Chen, Yongqiang Cheng, James |
| author_facet | Wang, Jiaqi Zhou, Yuhang Zhang, Zhixiong Chen, Qiguang Chen, Yongqiang Cheng, James |
| contents | Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12413 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization Wang, Jiaqi Zhou, Yuhang Zhang, Zhixiong Chen, Qiguang Chen, Yongqiang Cheng, James Machine Learning Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL. |
| title | DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.12413 |