Cover Image

Knowledge Distillation in Vision Transformers: A Critical Review

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Habib, Gousia, Saleem, Tausifa Jan, Lall, Brejesh
Format:	Preprint
Published:	2023
Subjects:	Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Comprehensive Review of Knowledge Distillation in Computer Vision
by: Habib, Gousia, et al.
Published: (2024)

LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression
by: Habib, Gousia, et al.
Published: (2023)

Optimizing Vision Transformers with Data-Free Knowledge Transfer
by: Habib, Gousia, et al.
Published: (2024)

Leveraging band diversity for feature selection in EO data
by: Hussain, Sadia, et al.
Published: (2025)

Single Stage Warped Cloth Learning and Semantic-Contextual Attention Feature Fusion for Virtual TryOn
by: Pathak, Sanhita, et al.
Published: (2023)

GraVITON: Graph based garment warping with attention guided inversion for Virtual-tryon
by: Pathak, Sanhita, et al.
Published: (2024)

DiffSTR: Controlled Diffusion Models for Scene Text Removal
by: Pathak, Sanhita, et al.
Published: (2024)

Deep Learning-Based Automated Segmentation of Uterine Myomas
by: Saleem, Tausifa Jan, et al.
Published: (2025)

Stride-Net: Fairness-Aware Disentangled Representation Learning for Chest X-Ray Diagnosis
by: Rashid, Darakshan, et al.
Published: (2026)

Harnessing The Power of Attention For Patch-Based Biomedical Image Classification
by: Habib, Gousia, et al.
Published: (2024)

Unified Multi-Dataset Training for TBPS
by: Chatterjee, Nilanjana, et al.
Published: (2026)

Dual Thinking and Logical Processing -- Are Multi-modal Large Language Models Closing the Gap with Human Vision ?
by: Dayanandan, Kailas, et al.
Published: (2024)

Continual Segmentation under Joint Nonstationarity
by: Pandey, Prashant, et al.
Published: (2026)

DuPLUS: Dual-Prompt Vision-Language Framework for Universal Medical Image Segmentation and Prognosis
by: Saeed, Numan, et al.
Published: (2025)

MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
by: Shaaban, Mai A., et al.
Published: (2025)

MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
by: Shakeel, Rozain, et al.
Published: (2026)

A Comprehensive Survey on Synthetic Infrared Image synthesis
by: Upadhyay, Avinash, et al.
Published: (2024)

Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification
by: Habib, Gousia, et al.
Published: (2024)

The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers
by: Son, Seungwoo, et al.
Published: (2023)

Vision Transformers with Self-Distilled Registers
by: Chen, Yinjie, et al.
Published: (2025)

A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition
by: Rahman, Dewan Tauhid, et al.
Published: (2025)

Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers
by: Tian, Huiyuan, et al.
Published: (2025)

Knowledge Distillation via the Target-aware Transformer
by: Lin, Sihao, et al.
Published: (2022)

Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning
by: Saleem, Tausifa Jan, et al.
Published: (2024)

KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling
by: Wang, Yu, et al.
Published: (2022)

Knowledge Distillation via Query Selection for Detection Transformer
by: Liu, Yi, et al.
Published: (2024)

Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge
by: Violos, John, et al.
Published: (2024)

Distilling Vision Transformers for Distortion-Robust Representation Learning
by: Alexis, Konstantinos, et al.
Published: (2026)

Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models
by: Zhang, Tiezheng, et al.
Published: (2025)

MambaVision: A Hybrid Mamba-Transformer Vision Backbone
by: Hatamizadeh, Ali, et al.
Published: (2024)

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
by: Zhang, Wenbo, et al.
Published: (2024)

MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation
by: Zhu, Junyou, et al.
Published: (2024)

Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
by: Sun, Haoyi, et al.
Published: (2026)

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation
by: Lv, Chonghua, et al.
Published: (2026)

Computer Vision-Based Early Detection of Container Loss at Sea
by: Lall, Vishakha, et al.
Published: (2026)

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
by: Wu, Size, et al.
Published: (2023)

LoRA-Enhanced Vision Transformer for Single Image based Morphing Attack Detection via Knowledge Distillation from EfficientNet
by: Shekhawat, Ria, et al.
Published: (2025)

Learning to Adapt to Position Bias in Vision Transformer Classifiers
by: Bruintjes, Robert-Jan, et al.
Published: (2025)

Knowledge Distillation Based on Transformed Teacher Matching
by: Zheng, Kaixiang, et al.
Published: (2024)

Analyzing Transformer Models and Knowledge Distillation Approaches for Image Captioning on Edge AI
by: Kwok, Wing Man Casca, et al.
Published: (2025)