Cover Image

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hao, Jitai, Huang, Qiang, Liu, Hao, Xiao, Xinyan, Ren, Zhaochun, Yu, Jun
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter
by: Hao, Jitai, et al.
Published: (2024)

Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
by: Hao, Jitai, et al.
Published: (2025)

DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity
by: Hao, Jitai, et al.
Published: (2026)

SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
by: Huang, Haiduo, et al.
Published: (2025)

EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
by: Zhang, Hao, et al.
Published: (2026)

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2026)

A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
by: Lu, Jinghui, et al.
Published: (2024)

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
by: Huang, Hongzhi, et al.
Published: (2025)

LLM-Oriented Token-Adaptive Knowledge Distillation
by: Xie, Xurong, et al.
Published: (2025)

ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers
by: Hu, Junyan, et al.
Published: (2024)

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
by: Li, Shiwei, et al.
Published: (2025)

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing
by: Chai, Ziwei, et al.
Published: (2024)

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
by: Chen, Yijie, et al.
Published: (2025)

Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation
by: Wei, Jingxuan, et al.
Published: (2024)

Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models
by: Cui, Xiao, et al.
Published: (2024)

TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching
by: Zeng, Runjia, et al.
Published: (2026)

A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens
by: Nie, Zhijie, et al.
Published: (2024)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
by: Shin, Seungjun, et al.
Published: (2025)

Lossless Token Sequence Compression via Meta-Tokens
by: Harvill, John, et al.
Published: (2025)

Token Distillation: Attention-aware Input Embeddings For New Tokens
by: Dobler, Konstantin, et al.
Published: (2025)

Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
by: Kim, Minsang, et al.
Published: (2026)

Disentangling Knowledge Representations for Large Language Model Editing
by: Zhang, Mengqi, et al.
Published: (2025)

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning
by: Ren, Pengjie, et al.
Published: (2024)

Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation
by: Zhang, Xiao, et al.
Published: (2025)

Cross-Tokenizer LLM Distillation through a Byte-Level Interface
by: Singh, Avyav Kumar, et al.
Published: (2026)

CTPD: Cross Tokenizer Preference Distillation
by: Nguyen, Truong, et al.
Published: (2026)

State over Tokens: Characterizing the Role of Reasoning Tokens
by: Levy, Mosh, et al.
Published: (2025)

Incorporating Domain Knowledge into Materials Tokenization
by: Oh, Yerim, et al.
Published: (2025)

Learning Compact Vision Tokens for Efficient Large Multimodal Models
by: Tang, Hao, et al.
Published: (2025)

Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
by: Liu, Hanbing, et al.
Published: (2025)

ETT-CKGE: Efficient Task-driven Tokens for Continual Knowledge Graph Embedding
by: Zhu, Lijing, et al.
Published: (2025)

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
by: Zhu, Mingcheng, et al.
Published: (2026)

ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation
by: Liu, Sicong, et al.
Published: (2026)

ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation
by: Ji, Yuelyu, et al.
Published: (2024)

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
by: Boizard, Nicolas, et al.
Published: (2024)

DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation
by: Vu, Duc Trung, et al.
Published: (2026)

Self-Distillation for Multi-Token Prediction
by: Zhao, Guoliang, et al.
Published: (2026)

Multi-Token Prediction via Self-Distillation
by: Kirchenbauer, John, et al.
Published: (2026)

Distilling Token-Trained Models into Byte-Level Models
by: Bao, Zishuo, et al.
Published: (2026)