Image de couverture de livre

LaDiMo: Layer-wise Distillation Inspired MoEfier

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Kim, Sungyoon, Kim, Youngjun, Moon, Kihyo, Jang, Minsung
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

ToDi: Token-wise Distillation via Fine-Grained Divergence Control
par: Jung, Seongryong, et autres
Publié: (2025)

DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
par: Yang, Mingyu, et autres
Publié: (2025)

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
par: Li, Zichong, et autres
Publié: (2025)

MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
par: Gurioli, Andrea, et autres
Publié: (2025)

On the Effect of Uncertainty on Layer-wise Inference Dynamics
par: Kim, Sunwoo, et autres
Publié: (2025)

SEDD: Scalable and Efficient Dataset Deduplication with GPUs
par: Son, Youngjun, et autres
Publié: (2025)

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models
par: Byun, Hoyoon, et autres
Publié: (2025)

Single LLM Debate, MoLaCE: Mixture of Latent Concept Experts Against Confirmation Bias
par: Kim, Hazel, et autres
Publié: (2025)

Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
par: Kovalev, Grigory, et autres
Publié: (2025)

MoFE: Mixture of Frozen Experts Architecture
par: Seo, Jean, et autres
Publié: (2025)

Fine-grained Gender Control in Machine Translation with Large Language Models
par: Lee, Minwoo, et autres
Publié: (2024)

Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset
par: Yu, Hyojeong, et autres
Publié: (2026)

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
par: Kim, Sungnyun, et autres
Publié: (2025)

FENCE: A Financial and Multimodal Jailbreak Detection Dataset
par: Kim, Mirae, et autres
Publié: (2026)

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios
par: Lee, Yunseung, et autres
Publié: (2026)

STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models
par: Jang, Kangwook, et autres
Publié: (2023)

MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
par: Chen, Yang, et autres
Publié: (2024)

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
par: Bae, Sangmin, et autres
Publié: (2024)

STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models
par: Lee, Kyumin, et autres
Publié: (2025)

MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning
par: Yang, Kang, et autres
Publié: (2025)

Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation
par: Yang, Runyan, et autres
Publié: (2025)

Layer-wise Swapping for Generalizable Multilingual Safety
par: Shin, Hyunseo, et autres
Publié: (2026)

Block-wise Codeword Embedding for Reliable Multi-bit Text Watermarking
par: Kim, Joeun, et autres
Publié: (2026)

SeDi-Instruct: Enhancing Alignment of Language Models through Self-Directed Instruction Generation
par: Kim, Jungwoo, et autres
Publié: (2025)

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
par: Singh, Navan Preet, et autres
Publié: (2026)

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
par: Kim, Gyeongman, et autres
Publié: (2024)

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
par: Wu, Haoyuan, et autres
Publié: (2025)

MoMoE: Mixture of Moderation Experts Framework for AI-Assisted Online Governance
par: Goyal, Agam, et autres
Publié: (2025)

FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving
par: Bin, Kyungmin, et autres
Publié: (2025)

How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Models
par: Kim, Minsung, et autres
Publié: (2025)

FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge
par: Yang, Nakyeong, et autres
Publié: (2025)

MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
par: Teo, Rachel S. Y., et autres
Publié: (2025)

Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
par: Tang, Yehui, et autres
Publié: (2025)

Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
par: Li, Pingzhi, et autres
Publié: (2025)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
par: Cao, Mingyu, et autres
Publié: (2024)

DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation
par: Maekawa, Aru, et autres
Publié: (2024)

Unplug and Play Language Models: Decomposing Experts in Language Models at Inference Time
par: Yang, Nakyeong, et autres
Publié: (2024)

ExpGuard: LLM Content Moderation in Specialized Domains
par: Choi, Minseok, et autres
Publié: (2026)

LLM Agents at the Roundtable: A Multi-Perspective and Dialectical Reasoning Framework for Essay Scoring
par: Jang, Jinhee, et autres
Publié: (2025)

Preference Distillation via Value based Reinforcement Learning
par: Kwon, Minchan, et autres
Publié: (2025)