Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.05088 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914906908917760 |
|---|---|
| author | Sariyildiz, Mert Bulent Weinzaepfel, Philippe Lucas, Thomas Larlus, Diane Kalantidis, Yannis |
| author_facet | Sariyildiz, Mert Bulent Weinzaepfel, Philippe Lucas, Thomas Larlus, Diane Kalantidis, Yannis |
| contents | Pretrained models have become a commodity and offer strong results on a broad range of tasks. In this work, we focus on classification and seek to learn a unique encoder able to take from several complementary pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers' influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task.
Project page and code: https://europe.naverlabs.com/unic |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05088 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UNIC: Universal Classification Models via Multi-teacher Distillation Sariyildiz, Mert Bulent Weinzaepfel, Philippe Lucas, Thomas Larlus, Diane Kalantidis, Yannis Computer Vision and Pattern Recognition Pretrained models have become a commodity and offer strong results on a broad range of tasks. In this work, we focus on classification and seek to learn a unique encoder able to take from several complementary pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers' influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task. Project page and code: https://europe.naverlabs.com/unic |
| title | UNIC: Universal Classification Models via Multi-teacher Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.05088 |