Cover Image

Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Anshumann, Zaidi, Mohd Abbas, Kedia, Akhil, Ahn, Jinwoo, Kwon, Taehwak, Lee, Kangwook, Lee, Haejun, Lee, Joohyung
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language 68T50 I.2.7
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
by: Tessema, Bethel Melesse, et al.
Published: (2024)

Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation
by: Jin, Heegon, et al.
Published: (2024)

Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning
by: Seo, Yeongbin, et al.
Published: (2024)

Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models
by: Kedia, Akhil, et al.
Published: (2024)

Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models
by: Park, Seungcheol, et al.
Published: (2025)

Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?
by: Pan, Leyi, et al.
Published: (2025)

RV-HATE: Reinforced Multi-Module Voting for Implicit Hate Speech Detection
by: Lee, Yejin, et al.
Published: (2025)

Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial
by: Johnson, Warren, et al.
Published: (2026)

Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information
by: Park, Seungcheol, et al.
Published: (2025)

Adaptive Steering and Remasking for Safe Generation in Diffusion Language Models
by: Lee, Yejin, et al.
Published: (2026)

Quantifying Genuine Awareness in Hallucination Prediction Beyond Question-Side Shortcuts
by: Seo, Yeongbin, et al.
Published: (2025)

Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom
by: Sen, Rishika, et al.
Published: (2025)

SentiCSE: A Sentiment-aware Contrastive Sentence Embedding Framework with Sentiment-guided Textual Similarity
by: Kim, Jaemin, et al.
Published: (2024)

Logits-Constrained Framework with RoBERTa for Ancient Chinese NER
by: Hua, Wenjie, et al.
Published: (2025)

Distilling Large Language Models for Efficient Clinical Information Extraction
by: Vedula, Karthik S., et al.
Published: (2024)

Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation
by: Vasilev, Stefan, et al.
Published: (2025)

Can LLMs Compute with Reasons?
by: Sandilya, Harshit, et al.
Published: (2024)

Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
by: Liu, Aiwei, et al.
Published: (2024)

LLMs for Legal Subsumption in German Employment Contracts
by: Wardas, Oliver, et al.
Published: (2025)

AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection
by: Lee, Yejin, et al.
Published: (2025)

Logits of API-Protected LLMs Leak Proprietary Information
by: Finlayson, Matthew, et al.
Published: (2024)

From Noise to Diversity: Random Embedding Injection in LLM Reasoning
by: Kim, Heejun, et al.
Published: (2026)

Efficient Adaptive Rejection Sampling for Accelerating Speculative Decoding in Large Language Models
by: Sun, Chendong, et al.
Published: (2025)

Synthesizing Behaviorally-Grounded Reasoning Chains: A Data-Generation Framework for Personal Finance LLMs
by: Theerthala, Akhil
Published: (2025)

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
by: Balter, Samuel G., et al.
Published: (2026)

When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
by: Wang, Yongjie, et al.
Published: (2025)

The Unlikely Duel: Evaluating Creative Writing in LLMs through a Unique Scenario
by: Gómez-Rodríguez, Carlos, et al.
Published: (2024)

TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
by: Liu, Aiwei, et al.
Published: (2024)

Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey
by: Fang, Xi, et al.
Published: (2024)

SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization
by: Lee, Wooin, et al.
Published: (2026)

How much do LLMs learn from negative examples?
by: Hamdan, Shadi, et al.
Published: (2025)

Communicative Agents for Slideshow Storytelling Video Generation based on LLMs
by: Fan, Jingxing, et al.
Published: (2025)

A Knowledge Enhanced Learning and Semantic Composition Model for Multi-Claim Fact Checking
by: Wang, Shuai, et al.
Published: (2021)

Improving Commonsense Bias Classification by Mitigating the Influence of Demographic Terms
by: Lee, JinKyu, et al.
Published: (2024)

K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology
by: Kim, Soyeon, et al.
Published: (2026)

A Comprehensive Survey of Compression Algorithms for Language Models
by: Park, Seungcheol, et al.
Published: (2024)

An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News
by: Consoli, Sergio, et al.
Published: (2025)

Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off
by: Zhao, Mingkuan, et al.
Published: (2025)

Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks
by: Lee, Hokyung, et al.
Published: (2024)

Prompt Engineering and the Effectiveness of Large Language Models in Enhancing Human Productivity
by: Anam, Rizal Khoirul
Published: (2025)