Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.02237 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917555466141696 |
|---|---|
| author | Li, Shucheng Jones, Iolo Tong, Alexander Bronstein, Michael M. |
| author_facet | Li, Shucheng Jones, Iolo Tong, Alexander Bronstein, Michael M. |
| contents | Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales. In principle, such distribution-level supervision remains agnostic to specific noise-data pairings of the teacher; this provides the student the freedom to remap latent noise, a behavior consistently observed in low-dimensional settings. Surprisingly, we find that in high-dimensional settings, distilled students spontaneously reproduce the original noise-data pairings of the teacher, a phenomenon we term copying. We demonstrate that copying is neither a byproduct of adversarial objectives nor a result of teacher memorization. Instead, our evidence suggests that copying is an emergent property arising from the limited geometric freedom of the student model during high-dimensional distillation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_02237 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation Li, Shucheng Jones, Iolo Tong, Alexander Bronstein, Michael M. Machine Learning Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales. In principle, such distribution-level supervision remains agnostic to specific noise-data pairings of the teacher; this provides the student the freedom to remap latent noise, a behavior consistently observed in low-dimensional settings. Surprisingly, we find that in high-dimensional settings, distilled students spontaneously reproduce the original noise-data pairings of the teacher, a phenomenon we term copying. We demonstrate that copying is neither a byproduct of adversarial objectives nor a result of teacher memorization. Instead, our evidence suggests that copying is an emergent property arising from the limited geometric freedom of the student model during high-dimensional distillation. |
| title | Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2606.02237 |