Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2604.03192 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917383041449984 |
|---|---|
| author | Sumit, Dipto Roy, Ankan Kumar Rodela, Sadia Khair Asha, Atia Haque Afrin, Mourchona Farhan, Niloy Sadeque, Farig Yousuf |
| author_facet | Sumit, Dipto Roy, Ankan Kumar Rodela, Sadia Khair Asha, Atia Haque Afrin, Mourchona Farhan, Niloy Sadeque, Farig Yousuf |
| contents | We study multiteacher knowledge distillation for low resource abstractive summarization from a reliability aware perspective. We introduce EWAD (Entropy Weighted Agreement Aware Distillation), a token level mechanism that routes supervision between teacher distillation and gold supervision based on inter teacher agreement, and CPDP (Capacity Proportional Divergence Preservation), a geometric constraint on the student position relative to heterogeneous teachers. Across two Bangla datasets, 13 BanglaT5 ablations, and eight Qwen2.5 experiments, we find that logit level KD provides the most reliable gains, while more complex distillation improves semantic similarity for short summaries but degrades longer outputs. Cross lingual pseudo label KD across ten languages retains 71-122 percent of teacher ROUGE L at 3.2x compression. A human validated multi judge LLM evaluation further reveals calibration bias in single judge pipelines. Overall, our results show that reliability aware distillation helps characterize when multi teacher supervision improves summarization and when data scaling outweighs loss engineering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03192 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization Sumit, Dipto Roy, Ankan Kumar Rodela, Sadia Khair Asha, Atia Haque Afrin, Mourchona Farhan, Niloy Sadeque, Farig Yousuf Computation and Language Artificial Intelligence We study multiteacher knowledge distillation for low resource abstractive summarization from a reliability aware perspective. We introduce EWAD (Entropy Weighted Agreement Aware Distillation), a token level mechanism that routes supervision between teacher distillation and gold supervision based on inter teacher agreement, and CPDP (Capacity Proportional Divergence Preservation), a geometric constraint on the student position relative to heterogeneous teachers. Across two Bangla datasets, 13 BanglaT5 ablations, and eight Qwen2.5 experiments, we find that logit level KD provides the most reliable gains, while more complex distillation improves semantic similarity for short summaries but degrades longer outputs. Cross lingual pseudo label KD across ten languages retains 71-122 percent of teacher ROUGE L at 3.2x compression. A human validated multi judge LLM evaluation further reveals calibration bias in single judge pipelines. Overall, our results show that reliability aware distillation helps characterize when multi teacher supervision improves summarization and when data scaling outweighs loss engineering. |
| title | Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.03192 |