Imagen de Portada

Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Pan, Leiyu, Su, Zhenpeng, Lv, Minxuan, Xiong, Yizhe, Zhang, Xiangwen, Lin, Zijia, Chen, Hui, Han, Jungong, Ding, Guiguang, Luo, Cheng, Zhang, Di, Gai, Kun, Xiong, Deyi
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
por: Lv, Minxuan, et al.
Publicado: (2025)

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts
por: Su, Zhenpeng, et al.
Publicado: (2024)

UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs
por: Xiong, Yizhe, et al.
Publicado: (2025)

CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
por: Su, Zhenpeng, et al.
Publicado: (2025)

Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
por: Su, Zhenpeng, et al.
Publicado: (2025)

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation
por: Xiong, Yizhe, et al.
Publicado: (2024)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
por: Lian, Haoran, et al.
Publicado: (2024)

An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation
por: Supryadi, et al.
Publicado: (2024)

Temporal Scaling Law for Large Language Models
por: Xiong, Yizhe, et al.
Publicado: (2024)

LANDeRMT: Detecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation
por: Zhu, Shaolin, et al.
Publicado: (2024)

MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts
por: Su, Zhenpeng, et al.
Publicado: (2024)

Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation
por: Xiong, Yizhe, et al.
Publicado: (2025)

Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
por: Su, Zhenpeng, et al.
Publicado: (2025)

Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs
por: Sun, Chenxi, et al.
Publicado: (2024)

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
por: Mei, Tiehua, et al.
Publicado: (2026)

Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
por: Liang, Yiwen, et al.
Publicado: (2025)

RepViT-SAM: Towards Real-Time Segmenting Anything
por: Wang, Ao, et al.
Publicado: (2023)

LSNet: See Large, Focus Small
por: Wang, Ao, et al.
Publicado: (2025)

RepViT: Revisiting Mobile CNN From ViT Perspective
por: Wang, Ao, et al.
Publicado: (2023)

MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
por: Su, Zhenpeng, et al.
Publicado: (2023)

Fast Quiet-STaR: Thinking Without Thought Tokens
por: Huang, Wei, et al.
Publicado: (2025)

CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs
por: Wang, Ao, et al.
Publicado: (2023)

[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
por: Wang, Ao, et al.
Publicado: (2024)

YOLOE: Real-Time Seeing Anything
por: Wang, Ao, et al.
Publicado: (2025)

Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts
por: Xu, Yangyang, et al.
Publicado: (2025)

LBPE: Long-token-first Tokenization to Improve Large Language Models
por: Lian, Haoran, et al.
Publicado: (2024)

Tracking and Segmenting Anything in Any Modality
por: Zhang, Tianlu, et al.
Publicado: (2025)

FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models
por: Liu, Yan, et al.
Publicado: (2024)

Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons
por: Leng, Yongqi, et al.
Publicado: (2024)

Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence
por: Lyu, Mengyao, et al.
Publicado: (2024)

YOLOv10: Real-Time End-to-End Object Detection
por: Wang, Ao, et al.
Publicado: (2024)

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
por: Zhao, Ziyu, et al.
Publicado: (2026)

DVMap: Fine-Grained Pluralistic Value Alignment via High-Consensus Demographic-Value Mapping
por: Zhu, Pengyun, et al.
Publicado: (2026)

Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models
por: Lian, Haoran, et al.
Publicado: (2024)

PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
por: Wang, Ao, et al.
Publicado: (2024)

Spatial Lifting for Dense Prediction
por: Xu, Mingzhi, et al.
Publicado: (2025)

CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Models
por: Shi, Ling, et al.
Publicado: (2024)

Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection
por: Yang, Hui-Yue, et al.
Publicado: (2024)

Interpret and Control Dense Retrieval with Sparse Latent Features
por: Kang, Hao, et al.
Publicado: (2024)

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan
por: Yang, Lei, et al.
Publicado: (2025)