Cover Image

Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Guo, Chenqi, Zhong, Shiwei, Liu, Xiaofeng, Feng, Qianli, Ma, Yinglong
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computer Vision and Pattern Recognition Image and Video Processing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image Fusion
by: Hsu, Chih-Chung, et al.
Published: (2024)

GuideSR: Rethinking Guidance for One-Step High-Fidelity Diffusion-Based Super-Resolution
by: Arora, Aditya, et al.
Published: (2025)

Rethinking the Nested U-Net Approach: Enhancing Biomarker Segmentation with Attention Mechanisms and Multiscale Feature Fusion
by: Wazir, Saad, et al.
Published: (2025)

MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution
by: Jiang, Yuxuan, et al.
Published: (2024)

MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis
by: Vuong, Trinh Thi Le, et al.
Published: (2023)

RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment
by: Feng, Chen, et al.
Published: (2023)

CTSR: Controllable Fidelity-Realness Trade-off Distillation for Real-World Image Super Resolution
by: Li, Runyi, et al.
Published: (2025)

Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data
by: Xu, Ziwang, et al.
Published: (2025)

DPF-Net: Physical Imaging Model Embedded Data-Driven Underwater Image Enhancement
by: Mei, Han, et al.
Published: (2025)

LiRCDepth: Lightweight Radar-Camera Depth Estimation via Knowledge Distillation and Uncertainty Guidance
by: Sun, Huawei, et al.
Published: (2024)

Efficient Star Distillation Attention Network for Lightweight Image Super-Resolution
by: Hao, Fangwei, et al.
Published: (2025)

KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp Segmentation
by: Trinh, Quoc-Huy, et al.
Published: (2023)

Face to Cartoon Incremental Super-Resolution using Knowledge Distillation
by: Devkatte, Trinetra, et al.
Published: (2024)

Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
by: Bougourzi, Fares, et al.
Published: (2024)

Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention
by: Wazir, Saad, et al.
Published: (2025)

Patient-Specific Autoregressive Models for Organ Motion Prediction in Radiotherapy
by: Lai, Yuxiang, et al.
Published: (2025)

Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images
by: Zhang, Yongheng, et al.
Published: (2025)

Enhancing Content Representation for AR Image Quality Assessment Using Knowledge Distillation
by: Sekhri, Aymen, et al.
Published: (2024)

HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification
by: EL-Assiouti, Omar S., et al.
Published: (2024)

Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localization
by: Zhang, Keyang, et al.
Published: (2025)

Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Images
by: Wang, Zhikang, et al.
Published: (2024)

Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imaging
by: Tölle, Malte, et al.
Published: (2024)

Multi-Task Multi-Scale Contrastive Knowledge Distillation for Efficient Medical Image Segmentation
by: Biswas, Risab
Published: (2024)

Leveraging Depth Maps and Attention Mechanisms for Enhanced Image Inpainting
by: Park, Jin Hyun, et al.
Published: (2025)

Accelerating Diffusion for SAR-to-Optical Image Translation via Adversarial Consistency Distillation
by: Bai, Xinyu, et al.
Published: (2024)

Edge AI-Enabled Chicken Health Detection Based on Enhanced FCOS-Lite and Knowledge Distillation
by: Tong, Qiang, et al.
Published: (2024)

Highly Constrained Coded Aperture Imaging Systems Design Via a Knowledge Distillation Approach
by: Suarez-Rodriguez, Leon, et al.
Published: (2024)

MambaLiteSR: Image Super-Resolution with Low-Rank Mamba using Knowledge Distillation
by: Aalishah, Romina, et al.
Published: (2025)

Phaseformer: Phase-based Attention Mechanism for Underwater Image Restoration and Beyond
by: Khan, MD Raqib, et al.
Published: (2024)

Distilling High Diagnostic Value Patches for Whole Slide Image Classification Using Attention Mechanism
by: Nan, Tianhang, et al.
Published: (2024)

Explainable Deep Learning for Cataract Detection in Retinal Images: A Dual-Eye and Knowledge Distillation Approach
by: Soflaei, MohammadReza Abbaszadeh Bavil, et al.
Published: (2025)

REACT-KD: Region-Aware Cross-modal Topological Knowledge Distillation for Interpretable Medical Image Classification
by: Chen, Hongzhao, et al.
Published: (2025)

Dual Attention Driven Lumbar Magnetic Resonance Image Feature Enhancement and Automatic Diagnosis of Herniation
by: Zhang, Lingrui, et al.
Published: (2025)

NestedMorph: Enhancing Deformable Medical Image Registration with Nested Attention Mechanisms
by: Kumar, Gurucharan Marthi Krishna, et al.
Published: (2024)

U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening
by: Kim, Sungpyo, et al.
Published: (2024)

Nickel and Diming Your GAN: A Dual-Method Approach to Enhancing GAN Efficiency via Knowledge Distillation
by: Yeo, Sangyeop, et al.
Published: (2024)

Dino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentation
by: Li, Haoyue, et al.
Published: (2025)

DiT-JSCC: Rethinking Deep JSCC with Diffusion Transformers and Semantic Representations
by: Tan, Kailin, et al.
Published: (2026)

Efficient Leaf Disease Classification and Segmentation using Midpoint Normalization Technique and Attention Mechanism
by: Taufik, Enam Ahmed, et al.
Published: (2025)

TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation
by: Liu, Ruiping, et al.
Published: (2022)