Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911651305881600 |
|---|---|
| author | Fang, Hao Zhang, Tianyi Zhuang, Tianqu Kong, Jiawei Gao, Kuofeng Chen, Bin Zheng, Leqi Xia, Shu-Tao Xu, Ke |
| author_facet | Fang, Hao Zhang, Tianyi Zhuang, Tianqu Kong, Jiawei Gao, Kuofeng Chen, Bin Zheng, Leqi Xia, Shu-Tao Xu, Ke |
| contents | Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03396 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective Fang, Hao Zhang, Tianyi Zhuang, Tianqu Kong, Jiawei Gao, Kuofeng Chen, Bin Zheng, Leqi Xia, Shu-Tao Xu, Ke Computation and Language Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property. |
| title | Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.03396 |