Cover Image

On the Generalization of Knowledge Distillation: An Information-Theoretic View

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Bingying, He, Haiyun
Format:	Preprint
Published:	2026
Subjects:	Information Theory Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Information-Theoretic Generalization Bounds for Deep Neural Networks
by: He, Haiyun, et al.
Published: (2024)

Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach
by: He, Haiyun, et al.
Published: (2024)

On the Generalization for Transfer Learning: An Information-Theoretic Analysis
by: Wu, Xuetong, et al.
Published: (2022)

Distributional Information Embedding: A Framework for Multi-bit Watermarking
by: He, Haiyun, et al.
Published: (2025)

Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
by: Barnes, L. P., et al.
Published: (2022)

Information-Theoretic Generative Clustering of Documents
by: Du, Xin, et al.
Published: (2024)

Information-Theoretic Discrete Diffusion
by: Jeon, Moongyu, et al.
Published: (2025)

An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamics
by: Liu, Wenliang, et al.
Published: (2024)

Information Theoretic Perspective on Representation Learning
by: Pereg, Deborah, et al.
Published: (2026)

Information-Theoretic Proofs for Diffusion Sampling
by: Reeves, Galen, et al.
Published: (2025)

An Information Theoretic Perspective on Conformal Prediction
by: Correia, Alvaro H. C., et al.
Published: (2024)

An Information-Theoretic Analysis of Temporal GNNs
by: Farzaneh, Amirmohammad
Published: (2024)

An Information-Theoretic Analysis of In-Context Learning
by: Jeon, Hong Jun, et al.
Published: (2024)

Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
by: Liu, Yanxiao, et al.
Published: (2026)

Directed Information $γ$-covering: An Information-Theoretic Framework for Context Engineering
by: Huang, Hai
Published: (2025)

Information-Theoretic Framework for Understanding Modern Machine-Learning
by: Feder, Meir, et al.
Published: (2025)

Information Theoretic Learning for Diffusion Models with Warm Start
by: Shen, Yirong, et al.
Published: (2025)

Information-Theoretic Measures on Lattices for High-Order Interactions
by: Liu, Zhaolu, et al.
Published: (2024)

Unifying Information-Theoretic and Pair-Counting Clustering Similarity
by: Gates, Alexander J.
Published: (2025)

Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States
by: Wang, Ziqiao, et al.
Published: (2022)

A Theoretical Framework for Statistical Evaluability of Generative Models
by: Aiyer, Shashaank, et al.
Published: (2026)

Information Theoretic Guarantees For Policy Alignment In Large Language Models
by: Mroueh, Youssef
Published: (2024)

Information-Theoretic Thresholds for the Alignments of Partially Correlated Graphs
by: Huang, Dong, et al.
Published: (2024)

Forecastability as an Information-Theoretic Limit on Prediction
by: Catt, Peter Maurice
Published: (2026)

An Information-Theoretic Criterion for Efficient Data Synthesis
by: Li, Hanyu, et al.
Published: (2026)

Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
by: Clauw, Kenzo, et al.
Published: (2024)

Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspective
by: Langer, Carlotta, et al.
Published: (2024)

Theoretical Analysis of Submodular Information Measures for Targeted Data Subset Selection
by: Beck, Nathan, et al.
Published: (2024)

Information-Theoretic Minimax Regret Bounds for Reinforcement Learning based on Duality
by: Bongole, Raghav, et al.
Published: (2024)

InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis
by: Tang, Shiqin, et al.
Published: (2025)

On the Hardness of Unsupervised Domain Adaptation: Optimal Learners and Information-Theoretic Perspective
by: Dong, Zhiyi, et al.
Published: (2025)

Information-Theoretic Multi-Model Fusion for Target-Oriented Adaptive Sampling in Materials Design
by: Zhang, Yixuan, et al.
Published: (2026)

Distillation-Enabled Knowledge Alignment for Generative Semantic Communications of AIGC Images
by: Hu, Jingzhi, et al.
Published: (2025)

An Information-Theoretic Approach to Understanding Transformers' In-Context Learning of Variable-Order Markov Chains
by: Zhou, Ruida, et al.
Published: (2024)

On Causality in Domain Adaptation and Semi-Supervised Learning: an Information-Theoretic Analysis for Parametric Models
by: Wu, Xuetong, et al.
Published: (2022)

Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit
by: Zhang, Bohan, et al.
Published: (2025)

On Theoretical Limits of Learning with Label Differential Privacy
by: Zhao, Puning, et al.
Published: (2025)

ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
by: Bhattacharya, Sagnik, et al.
Published: (2025)

An Effective Information Theoretic Framework for Channel Pruning
by: Chen, Yihao, et al.
Published: (2024)

Machine Unlearning via Information Theoretic Regularization
by: Xu, Shizhou, et al.
Published: (2025)