Cover Image

What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry View

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Saadi, Khouloud, Wang, Di
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Validity-Calibrated Reasoning Distillation
by: Saadi, Khouloud, et al.
Published: (2026)

JGU Mainz's Submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: MT and QA
by: Saadi, Hossain Shaikh, et al.
Published: (2025)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
by: Yang, Junjie, et al.
Published: (2025)

Distilling Text Style Transfer With Self-Explanation From LLMs
by: Zhang, Chiyu, et al.
Published: (2024)

"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?
by: Reichman, Benjamin, et al.
Published: (2025)

What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
by: Staufer, Dimitri
Published: (2025)

What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents
by: Xue, Zhaoqian, et al.
Published: (2024)

CoMMET: To What Extent Can LLMs Perform Theory of Mind Tasks?
by: Chen, Ruirui, et al.
Published: (2026)

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
by: Datta, Joyeeta, et al.
Published: (2025)

Feature Structure Distillation with Centered Kernel Alignment in BERT Transferring
by: Jung, Hee-Jun, et al.
Published: (2022)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
by: Zhang, Xuemiao, et al.
Published: (2025)

Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs
by: Wu, Xinwei, et al.
Published: (2026)

Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance
by: Yin, Ziqi, et al.
Published: (2024)

Divide-or-Conquer? Which Part Should You Distill Your LLM?
by: Wu, Zhuofeng, et al.
Published: (2024)

Hybrid Policy Distillation for LLMs
by: Zhu, Wenhong, et al.
Published: (2026)

Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains
by: Wang, Pingjie, et al.
Published: (2025)

LLMs Should Express Uncertainty Explicitly
by: Guo, Junyu, et al.
Published: (2026)

The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
by: Song, Yifan, et al.
Published: (2024)

Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation
by: Raha, Tathagata, et al.
Published: (2026)

LLMs Should Incorporate Explicit Mechanisms for Human Empathy
by: You, Xiaoxing, et al.
Published: (2026)

Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks
by: Bao, Yuntai, et al.
Published: (2025)

Bridging Language Barriers in Healthcare: A Study on Arabic LLMs
by: Saadi, Nada, et al.
Published: (2025)

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
by: Boizard, Nicolas, et al.
Published: (2024)

Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
by: Feng, Kaituo, et al.
Published: (2024)

Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
by: Tian, Yijun, et al.
Published: (2024)

Exploring the Personality Traits of LLMs through Latent Features Steering
by: Yang, Shu, et al.
Published: (2024)

The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
by: Tang, Zhenheng, et al.
Published: (2025)

DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
by: Chen, Jennifer, et al.
Published: (2025)

Retrieval Augmented Question Answering: When Should LLMs Admit Ignorance?
by: Wang, Dingmin, et al.
Published: (2025)

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
by: Du, Dayou, et al.
Published: (2024)

Dense X Retrieval: What Retrieval Granularity Should We Use?
by: Chen, Tong, et al.
Published: (2023)

Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning
by: Yue, Yuanhao, et al.
Published: (2024)

Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models
by: Wang, Shuo, et al.
Published: (2024)

Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation
by: Klimaszewski, Mateusz, et al.
Published: (2024)

Training Task Experts through Retrieval Based Distillation
by: Ge, Jiaxin, et al.
Published: (2024)

What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs
by: Yan, Xinlan, et al.
Published: (2025)

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks
by: Kirch, Nathalie, et al.
Published: (2024)

Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation
by: Zhang, Xiao, et al.
Published: (2025)

SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition
by: Wu, Mengsong, et al.
Published: (2025)

LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information
by: van Buuren, Stef
Published: (2026)