Learning Task-Agnostic Representations through Multi-Teacher Distillation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915567719415808 |
|---|---|
| author | Formont, Philippe Darrin, Maxime Karimian, Banafsheh Cheung, Jackie CK Granger, Eric Ayed, Ismail Ben Shateri, Mohammadhadi Piantanida, Pablo |
| author_facet | Formont, Philippe Darrin, Maxime Karimian, Banafsheh Cheung, Jackie CK Granger, Eric Ayed, Ismail Ben Shateri, Mohammadhadi Piantanida, Pablo |
| contents | Casting complex inputs into tractable representations is a critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. In this paper, we introduce a task-agnostic framework based on a ``majority vote" objective function. We demonstrate that this function is bounded by the mutual information between student and teachers' embeddings, leading to a task-agnostic distillation loss that eliminates dependence on task-specific labels or prior knowledge. Our evaluations across text, vision models, and molecular modeling show that our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such as classification, clustering, or regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_18680 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning Task-Agnostic Representations through Multi-Teacher Distillation Formont, Philippe Darrin, Maxime Karimian, Banafsheh Cheung, Jackie CK Granger, Eric Ayed, Ismail Ben Shateri, Mohammadhadi Piantanida, Pablo Machine Learning Casting complex inputs into tractable representations is a critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. In this paper, we introduce a task-agnostic framework based on a ``majority vote" objective function. We demonstrate that this function is bounded by the mutual information between student and teachers' embeddings, leading to a task-agnostic distillation loss that eliminates dependence on task-specific labels or prior knowledge. Our evaluations across text, vision models, and molecular modeling show that our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such as classification, clustering, or regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities. |
| title | Learning Task-Agnostic Representations through Multi-Teacher Distillation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.18680 |