Quantifying Knowledge Distillation Using Partial Information Decomposition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908300583370752 |
|---|---|
| author | Dissanayake, Pasan Hamman, Faisal Halder, Barproda Sucholutsky, Ilia Zhang, Qiuyi Dutta, Sanghamitra |
| author_facet | Dissanayake, Pasan Hamman, Faisal Halder, Barproda Sucholutsky, Ilia Zhang, Qiuyi Dutta, Sanghamitra |
| contents | Knowledge distillation deploys complex machine learning models in resource-constrained environments by training a smaller student model to emulate internal representations of a complex teacher model. However, the teacher's representations can also encode nuisance or additional information not relevant to the downstream task. Distilling such irrelevant information can actually impede the performance of a capacity-limited student model. This observation motivates our primary question: What are the information-theoretic limits of knowledge distillation? To this end, we leverage Partial Information Decomposition to quantify and explain the transferred knowledge and knowledge left to distill for a downstream task. We theoretically demonstrate that the task-relevant transferred knowledge is succinctly captured by the measure of redundant information about the task between the teacher and student. We propose a novel multi-level optimization to incorporate redundant information as a regularizer, leading to our framework of Redundant Information Distillation (RID). RID leads to more resilient and effective distillation under nuisance teachers as it succinctly quantifies task-relevant knowledge rather than simply aligning student and teacher representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07483 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quantifying Knowledge Distillation Using Partial Information Decomposition Dissanayake, Pasan Hamman, Faisal Halder, Barproda Sucholutsky, Ilia Zhang, Qiuyi Dutta, Sanghamitra Machine Learning Computer Vision and Pattern Recognition Information Theory Image and Video Processing Knowledge distillation deploys complex machine learning models in resource-constrained environments by training a smaller student model to emulate internal representations of a complex teacher model. However, the teacher's representations can also encode nuisance or additional information not relevant to the downstream task. Distilling such irrelevant information can actually impede the performance of a capacity-limited student model. This observation motivates our primary question: What are the information-theoretic limits of knowledge distillation? To this end, we leverage Partial Information Decomposition to quantify and explain the transferred knowledge and knowledge left to distill for a downstream task. We theoretically demonstrate that the task-relevant transferred knowledge is succinctly captured by the measure of redundant information about the task between the teacher and student. We propose a novel multi-level optimization to incorporate redundant information as a regularizer, leading to our framework of Redundant Information Distillation (RID). RID leads to more resilient and effective distillation under nuisance teachers as it succinctly quantifies task-relevant knowledge rather than simply aligning student and teacher representations. |
| title | Quantifying Knowledge Distillation Using Partial Information Decomposition |
| topic | Machine Learning Computer Vision and Pattern Recognition Information Theory Image and Video Processing |
| url | https://arxiv.org/abs/2411.07483 |