Cover Image

ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Junzhang, Liu, Tingkai, Sui, Yueyuan, Xia, Stephen
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms
by: Sui, Yueyuan, et al.
Published: (2024)

R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
by: Song, Sanghyeob, et al.
Published: (2026)

Reinforcement Learning via Self-Distillation
by: Hübotter, Jonas, et al.
Published: (2026)

MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs
by: Xu, Hongjun, et al.
Published: (2025)

VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
by: Zhou, Jingbo, et al.
Published: (2026)

Context-Former: Stitching via Latent Conditioned Sequence Modeling
by: Zhang, Ziqi, et al.
Published: (2024)

RecurFormer: Not All Transformer Heads Need Self-Attention
by: Yan, Ruiqing, et al.
Published: (2024)

KuaiFormer: Transformer-Based Retrieval at Kuaishou
by: Liu, Chi, et al.
Published: (2024)

$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
by: Du, Zhenbang, et al.
Published: (2026)

CellCLAT: Preserving Topology and Trimming Redundancy in Self-Supervised Cellular Contrastive Learning
by: Qin, Bin, et al.
Published: (2025)

GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers
by: Ai, Guoguo, et al.
Published: (2024)

NoiseFormer -- Noise Diffused Symmetric Attention Transformer
by: Kumar, Phani, et al.
Published: (2026)

HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
by: Ding, Ken
Published: (2026)

Multilingual Safety Alignment via Self-Distillation
by: Qin, Ruiyang, et al.
Published: (2026)

OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions
by: Luo, Charles
Published: (2026)

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures
by: Gao, Yihang, et al.
Published: (2024)

Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation
by: Silvestri, Gianluigi, et al.
Published: (2026)

GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
by: Tang, Xiaohang, et al.
Published: (2026)

FedSDR: Federated Self-Distillation with Rectification
by: Ren, Ziheng, et al.
Published: (2026)

DavIR: Data Selection via Implicit Reward for Large Language Models
by: Zhou, Haotian, et al.
Published: (2023)

ShapeFormer: Shapelet Transformer for Multivariate Time Series Classification
by: Le, Xuan-May, et al.
Published: (2024)

RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals
by: Heo, Jaemu, et al.
Published: (2025)

Self-Distilled Disentangled Learning for Counterfactual Prediction
by: Li, Xinshu, et al.
Published: (2024)

Low-redundancy Distillation for Continual Learning
by: Liu, RuiQi, et al.
Published: (2023)

ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
by: Chen, Yuqi, et al.
Published: (2024)

StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel
by: Cutler, Dylan, et al.
Published: (2025)

Dataset Distillation for Offline Reinforcement Learning
by: Light, Jonathan, et al.
Published: (2024)

The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
by: Qasim, Kaleem Ullah, et al.
Published: (2026)

VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation
by: Corn, Eli, et al.
Published: (2026)

ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling
by: Zhang, Yuanyun, et al.
Published: (2025)

BioFormer: Rethinking Cross-Subject Generalization via Spectral Structural Alignment in Biomedical Time-Series
by: Du, Guikang, et al.
Published: (2026)

Time Series Representation Learning with Supervised Contrastive Temporal Transformer
by: Liu, Yuansan, et al.
Published: (2024)

VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing
by: Wang, Yixiao, et al.
Published: (2025)

Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
by: Liu, Zehao, et al.
Published: (2026)

Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
by: He, Yuting, et al.
Published: (2025)

Personalized Federated Learning via Backbone Self-Distillation
by: Wang, Pengju, et al.
Published: (2024)

FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation
by: Yang, Yingguang, et al.
Published: (2026)

When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
by: Liu, Xiaogeng, et al.
Published: (2026)

PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence
by: Xu, Yuanda, et al.
Published: (2026)

How Transformers Learn to Plan via Multi-Token Prediction
by: Huang, Jianhao, et al.
Published: (2026)