Cover Image

DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ko, Jongwoo, Chen, Tianyi, Kim, Sungnyun, Ding, Tianyu, Liang, Luming, Zharkov, Ilya, Yun, Se-Young
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DistiLLM: Towards Streamlined Distillation for Large Language Models
by: Ko, Jongwoo, et al.
Published: (2024)

Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
by: Ko, Jongwoo, et al.
Published: (2025)

S3Editor: A Sparse Semantic-Disentangled Self-Training Framework for Face Video Editing
by: Wang, Guangzhi, et al.
Published: (2024)

From Belief Entrenchment to Robust Reasoning in LLM Agents
by: Oh, Jihwan, et al.
Published: (2025)

DREAM: Diffusion Rectification and Estimation-Adaptive Models
by: Zhou, Jinxin, et al.
Published: (2023)

PerMix-RLVR: Preserving Persona Expressivity under Verifiable-Reward Alignment
by: Oh, Jihwan, et al.
Published: (2026)

FORA: Fast-Forward Caching in Diffusion Transformer Acceleration
by: Selvaraju, Pratheba, et al.
Published: (2024)

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
by: Ko, Jongwoo, et al.
Published: (2026)

Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking
by: Schlatt, Ferdinand, et al.
Published: (2024)

Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models
by: Yang, Yongjin, et al.
Published: (2023)

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
by: Ding, Tianyu, et al.
Published: (2023)

Cat-AIR: Content and Task-Aware All-in-One Image Restoration
by: Jiang, Jiachen, et al.
Published: (2025)

AdaContour: Adaptive Contour Descriptor with Hierarchical Representation
by: Ding, Tianyu, et al.
Published: (2024)

CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering
by: Zhu, Haidong, et al.
Published: (2023)

HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning
by: Chen, Tianyi, et al.
Published: (2024)

Bayesian Principles Improve Prompt Learning In Vision-Language Models
by: Kim, Mingyu, et al.
Published: (2025)

FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning
by: Kim, Seongyoon, et al.
Published: (2024)

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition
by: Kim, Sungnyun, et al.
Published: (2024)

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
by: Kim, Sungnyun, et al.
Published: (2025)

Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation
by: Phan, Phuc, et al.
Published: (2024)

ProCrop: Learning Aesthetic Image Cropping from Professional Compositions
by: Zhang, Ke, et al.
Published: (2025)

WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference
by: Chen, Sihan, et al.
Published: (2025)

StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
by: Chen, Tianyi, et al.
Published: (2026)

Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses
by: Kim, Sungnyun, et al.
Published: (2025)

STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models
by: Jang, Kangwook, et al.
Published: (2023)

Scalable Frameworks for Real-World Audio-Visual Speech Recognition
by: Kim, Sungnyun
Published: (2025)

First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models
by: Ha, Jiwoo, et al.
Published: (2026)

Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
by: Yang, Yongjin, et al.
Published: (2025)

Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
by: Yang, Yongjin, et al.
Published: (2024)

CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Causal Significance and Consistency
by: Wang, Kangsheng, et al.
Published: (2024)

Motion Graph Unleashed: A Novel Approach to Video Prediction
by: Zhong, Yiqi, et al.
Published: (2024)

Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers
by: Ko, Youmin, et al.
Published: (2025)

TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
by: Kim, Taesoo, et al.
Published: (2025)

LLM Pruning and Distillation in Practice: The Minitron Approach
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2024)

Evolutionary Contrastive Distillation for Language Model Alignment
by: Katz-Samuels, Julian, et al.
Published: (2024)

Medal Matters: Probing LLMs' Failure Cases Through Olympic Rankings
by: Choi, Juhwan, et al.
Published: (2024)

SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data
by: Bae, Suyoung, et al.
Published: (2025)

Guiding Reasoning in Small Language Models with LLM Assistance
by: Kim, Yujin, et al.
Published: (2025)

Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation
by: Choi, Juhwan, et al.
Published: (2024)

Self-Training Elicits Concise Reasoning in Large Language Models
by: Munkhbat, Tergel, et al.
Published: (2025)