Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.13073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917842140528640 |
|---|---|
| author | Peng, Shuman Khoeini, Arash Vaswani, Sharan Ester, Martin |
| author_facet | Peng, Shuman Khoeini, Arash Vaswani, Sharan Ester, Martin |
| contents | The quality of self-supervised pre-trained embeddings on out-of-distribution (OOD) data is poor without fine-tuning. A straightforward and simple approach to improving the generalization of pre-trained representation to OOD data is the use of deep ensembles. However, obtaining an effective ensemble in the embedding space with only unlabeled data remains an unsolved problem. We first perform a theoretical analysis that reveals the relationship between individual hyperspherical embedding spaces in an ensemble. We then design a principled method to align these embedding spaces in an unsupervised manner. Experimental results on the MNIST dataset show that our embedding-space ensemble method improves pre-trained embedding quality on in-distribution and OOD data compared to single encoders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13073 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensembles Peng, Shuman Khoeini, Arash Vaswani, Sharan Ester, Martin Machine Learning Computer Vision and Pattern Recognition The quality of self-supervised pre-trained embeddings on out-of-distribution (OOD) data is poor without fine-tuning. A straightforward and simple approach to improving the generalization of pre-trained representation to OOD data is the use of deep ensembles. However, obtaining an effective ensemble in the embedding space with only unlabeled data remains an unsolved problem. We first perform a theoretical analysis that reveals the relationship between individual hyperspherical embedding spaces in an ensemble. We then design a principled method to align these embedding spaces in an unsupervised manner. Experimental results on the MNIST dataset show that our embedding-space ensemble method improves pre-trained embedding quality on in-distribution and OOD data compared to single encoders. |
| title | Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensembles |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.13073 |