Cover Image

Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dasgupta, Sayantan, Cohn, Trevor, Baldwin, Timothy
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval
by: Yang, Jinrui, et al.
Published: (2023)

Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain
by: Jang, Eugene, et al.
Published: (2024)

Don't Throw Away Data: Better Sequence Knowledge Distillation
by: Wang, Jun, et al.
Published: (2024)

Sample, Don't Search: Rethinking Test-Time Alignment for Language Models
by: Faria, Gonçalo, et al.
Published: (2025)

If You Don't Understand It, Don't Use It: Eliminating Trojans with Filters Between Layers
by: Hernandez, Adriano
Published: (2024)

Reasoning Models Don't Just Think Longer, They Move Differently
by: Gjølbye, Anders, et al.
Published: (2026)

I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
by: Cohen, Roi, et al.
Published: (2024)

Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
by: Tyukin, Georgy, et al.
Published: (2024)

Large Language Models Must Be Taught to Know What They Don't Know
by: Kapoor, Sanyam, et al.
Published: (2024)

Don't Walk the Line: Boundary Guidance for Filtered Generation
by: Ball, Sarah, et al.
Published: (2025)

Reasoning Models Don't Always Say What They Think
by: Chen, Yanda, et al.
Published: (2025)

Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations
by: Deng, Yang, et al.
Published: (2024)

Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming
by: Kavumba, Pride, et al.
Published: (2026)

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
by: Le, Dong, et al.
Published: (2026)

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
by: Roy, Debjyoti Saha, et al.
Published: (2024)

Zero-Shot Performance Prediction for Probabilistic Scaling Laws
by: Schram, Viktoria, et al.
Published: (2025)

Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
by: Hariri, Mohsen, et al.
Published: (2025)

Learn and Don't Forget: Adding a New Language to ASR Foundation Models
by: Qian, Mengjie, et al.
Published: (2024)

Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR
by: Bai, Junwen, et al.
Published: (2024)

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
by: Bandarkar, Lucas, et al.
Published: (2026)

Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation
by: Vernikos, Giorgos, et al.
Published: (2024)

Benchmarking Gender and Political Bias in Large Language Models
by: Yang, Jinrui, et al.
Published: (2025)

Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
by: Leemann, Tobias, et al.
Published: (2024)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
by: Cao, Mingyu, et al.
Published: (2024)

Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models
by: Zausinger, Jonas, et al.
Published: (2024)

Why Don't Prompt-Based Fairness Metrics Correlate?
by: Zayed, Abdelrahman, et al.
Published: (2024)

You Don't Need Prompt Engineering Anymore: The Prompting Inversion
by: Khan, Imran
Published: (2025)

Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalization
by: Zhou, Jin Peng, et al.
Published: (2024)

Don't Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention
by: Pi, Shu-Ting, et al.
Published: (2026)

Don't lie to your friends: Learning what you know from collaborative self-play
by: Eisenstein, Jacob, et al.
Published: (2025)

Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
by: Zheng, Lin, et al.
Published: (2026)

Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
by: Engländer, Leon, et al.
Published: (2026)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

Invariant Features in Language Models: Geometric Characterization and Model Attribution
by: Dasgupta, Agnibh, et al.
Published: (2026)

Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search
by: Fadeeva, Ekaterina, et al.
Published: (2025)

JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks
by: Kaneko, Masahiro, et al.
Published: (2026)

Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
by: Kovalev, Grigory, et al.
Published: (2025)

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models
by: Doering, Nigel, et al.
Published: (2024)

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs
by: Kaneko, Masahiro, et al.
Published: (2025)

Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?
by: Jedidi, Nour, et al.
Published: (2025)