Cover Image

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Datta, Joyeeta, Doll, Niclas, Ramadan, Qusai, Boukhers, Zeyd
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Falcon 7b for Software Mention Detection in Scholarly Documents
by: Khan, AmeerAli, et al.
Published: (2024)

Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding
by: Boukhers, Zeyd, et al.
Published: (2024)

Comparison of Feature Learning Methods for Metadata Extraction from PDF Scholarly Documents
by: Boukhers, Zeyd, et al.
Published: (2025)

Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspective
by: Kong, Lingxiao, et al.
Published: (2025)

EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuning
by: Kong, Lingxiao, et al.
Published: (2025)

Hybrid Physics and Deep Learning Model for Interpretable Vehicle State Prediction
by: Baier, Alexandra, et al.
Published: (2021)

Data Trading and Monetization: Challenges and Open Research Directions
by: Ramadan, Qusai, et al.
Published: (2024)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
by: Wu, Taiqiang, et al.
Published: (2023)

FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
by: Kirchenbauer, John, et al.
Published: (2025)

Universal Response and Emergence of Induction in LLMs
by: Luick, Niclas
Published: (2024)

Efficient Knowledge Injection in LLMs via Self-Distillation
by: Kujanpää, Kalle, et al.
Published: (2024)

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
by: Pan, Zhuoshi, et al.
Published: (2024)

Multilingual Open QA on the MIA Shared Task
by: Yarrabelly, Navya, et al.
Published: (2025)

Towards Systematic Specification and Verification of Fairness Requirements: A Position Paper
by: Ramadan, Qusai, et al.
Published: (2025)

Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in Product QA Agents
by: Lewis, Ashley, et al.
Published: (2025)

Integrating Domain Knowledge for Financial QA: A Multi-Retriever RAG Approach with LLMs
by: Zhang, Yukun, et al.
Published: (2025)

MiniDisc: Minimal Distillation Schedule for Language Model Compression
by: Zhang, Chen, et al.
Published: (2022)

WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
by: Thede, Lukas, et al.
Published: (2025)

A Survey on Symbolic Knowledge Distillation of Large Language Models
by: Acharya, Kamal, et al.
Published: (2024)

Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons
by: Leng, Yongqi, et al.
Published: (2024)

Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
by: Kovalev, Grigory, et al.
Published: (2025)

Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
by: Hamman, Faisal, et al.
Published: (2025)

Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
by: Xiao, Hanqi, et al.
Published: (2025)

LatentQA: Teaching LLMs to Decode Activations Into Natural Language
by: Pan, Alexander, et al.
Published: (2024)

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
by: Fan, Tao, et al.
Published: (2025)

Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
by: Fang, Luyang, et al.
Published: (2025)

Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs
by: Scarlatos, Alexander, et al.
Published: (2024)

Knowledge Distillation with Training Wheels
by: Liu, Guanlin, et al.
Published: (2025)

Knowledge Distillation from Large Language Models for Household Energy Modeling
by: Takrouri, Mohannad, et al.
Published: (2025)

PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams -- Dataset Construction and Evaluation
by: Carrillo-Larco, Rodrigo M., et al.
Published: (2025)

Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guarantees
by: Montreuil, Yannis, et al.
Published: (2024)

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
by: Liu, Chi, et al.
Published: (2026)

SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA
by: Xu, Haozhou, et al.
Published: (2025)

Learning to Correct for QA Reasoning with Black-box LLMs
by: Kim, Jaehyung, et al.
Published: (2024)

Agentic Adversarial QA for Improving Domain-Specific LLMs
by: Grari, Vincent, et al.
Published: (2026)

Sinkhorn Distance Minimization for Knowledge Distillation
by: Cui, Xiao, et al.
Published: (2024)

Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
by: Yang, Junjie, et al.
Published: (2025)

QA-Calibration of Language Model Confidence Scores
by: Manggala, Putra, et al.
Published: (2024)

Beyond Trading Data: The Hidden Influence of Public Awareness and Interest on Cryptocurrency Volatility
by: Boukhers, Zeyd, et al.
Published: (2022)