Cover Image

From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Nishu, Kumari, Mehta, Sachin, Abnar, Samira, Farajtabar, Mehrdad, Horton, Maxwell, Najibi, Mahyar, Nabi, Moin, Cho, Minsik, Naik, Devang
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
by: Samragh, Mohammad, et al.
Published: (2025)

SLiCK: Exploiting Subsequences for Length-Constrained Keyword Spotting
by: Nishu, Kumari, et al.
Published: (2024)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
by: Samragh, Mohammad, et al.
Published: (2024)

DOT-MoE: Differentiable Optimal Transport for MoEfication
by: Bamba, Udbhav, et al.
Published: (2026)

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data
by: Mehta, Sachin, et al.
Published: (2024)

LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
by: Fu, Qichen, et al.
Published: (2024)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
by: Hannah, Lauren. A, et al.
Published: (2025)

KV Prediction for Improved Time to First Token
by: Horton, Maxwell, et al.
Published: (2024)

M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference
by: Bhendawade, Nikhil, et al.
Published: (2025)

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
by: Cho, Minsik, et al.
Published: (2024)

CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning
by: Cao, Qingqing, et al.
Published: (2024)

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
by: Chegini, Atoosa, et al.
Published: (2024)

Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context
by: Alizadeh, Keivan, et al.
Published: (2026)

Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
by: Wen, Yuxin, et al.
Published: (2024)

Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
by: Alizadeh, Keivan, et al.
Published: (2024)

TIDE: Every Layer Knows the Token Beneath the Context
by: Jaiswal, Ajay, et al.
Published: (2026)

Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder
by: Bittar, Alexandre, et al.
Published: (2023)

MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
by: Zibakhsh, Soheil, et al.
Published: (2025)

SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
by: Shafipour, Rasoul, et al.
Published: (2024)

Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
by: Bhendawade, Nikhil, et al.
Published: (2025)

Computational Bottlenecks of Training Small-scale Large Language Models
by: Ashkboos, Saleh, et al.
Published: (2024)

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
by: Shojaee, Parshin, et al.
Published: (2025)

Off-diagonally symmetric alternating sign matrices
by: Kumari, Nishu
Published: (2025)

A determinantal formula for orthosymplectic Schur functions
by: Kumari, Nishu
Published: (2024)

Skew hook Schur functions and the cyclic sieving phenomenon
by: Kumari, Nishu
Published: (2022)

MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
by: Jaiswal, Ajay, et al.
Published: (2026)

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework
by: Mehta, Sachin, et al.
Published: (2024)

Bytes Are All You Need: Transformers Operating Directly On File Bytes
by: Horton, Maxwell, et al.
Published: (2023)

Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
by: Vemulapalli, Raviteja, et al.
Published: (2023)

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
by: Klein, Tassilo, et al.
Published: (2024)

Learning Private Representations through Entropy-based Adversarial Training
by: Klein, Tassilo, et al.
Published: (2025)

Limit profile for the transpose top-2 with random shuffle
by: Ghosh, Subhajit, et al.
Published: (2024)

Murnaghan--Nakayama rules for symplectic, orthogonal and orthosymplectic Schur functions
by: Kumari, Nishu, et al.
Published: (2024)

Further results for classical and universal characters twisted by roots of unity
by: Ayyer, Arvind, et al.
Published: (2024)

Diffusion Models as Masked Audio-Video Learners
by: Nunez, Elvis, et al.
Published: (2023)

LLM in a flash: Efficient Large Language Model Inference with Limited Memory
by: Alizadeh, Keivan, et al.
Published: (2023)

H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages
by: Zakershahrak, Mehrdad, et al.
Published: (2025)

TiC-CLIP: Continual Training of CLIP Models
by: Garg, Saurabh, et al.
Published: (2023)

Engineering the plant circadian clock for latitudinal adaptation as a strategy to secure agricultural productivity on a changing planet
by: Aisha Gerhardt, et al.
Published: (2025)

An Efficient and Streaming Audio Visual Active Speaker Detection System
by: Kundu, Arnav, et al.
Published: (2024)