Image de couverture de livre

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Dou, Shihan, Zhou, Enyu, Liu, Yan, Gao, Songyang, Zhao, Jun, Shen, Wei, Zhou, Yuhao, Xi, Zhiheng, Wang, Xiao, Fan, Xiaoran, Pu, Shiliang, Zhu, Jiang, Zheng, Rui, Gui, Tao, Zhang, Qi, Huang, Xuanjing
Format:	Preprint
Publié:	2023
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement
par: Xi, Zhiheng, et autres
Publié: (2023)

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
par: Dou, Shihan, et autres
Publié: (2024)

Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
par: Zhang, Zhihao, et autres
Publié: (2025)

Steering LLMs via Scalable Interactive Oversight
par: Zhou, Enyu, et autres
Publié: (2026)

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals
par: Zheng, Rui, et autres
Publié: (2024)

MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning
par: Lin, Jiahang, et autres
Publié: (2026)

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs
par: Zhou, Xin, et autres
Publié: (2024)

JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
par: Wang, Yuhui, et autres
Publié: (2026)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
par: Li, Junzhuo, et autres
Publié: (2025)

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
par: Manzoni, Andrea
Publié: (2026)

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
par: Chen, Anrui, et autres
Publié: (2026)

RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
par: Zhou, Enyu, et autres
Publié: (2024)

Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing
par: Liu, Wentao, et autres
Publié: (2025)

ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
par: Yang, Yuchen, et autres
Publié: (2026)

FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
par: Zhang, Boyang, et autres
Publié: (2025)

Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
par: Wang, Zhaokun, et autres
Publié: (2025)

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
par: Jiang, Changhao, et autres
Publié: (2025)

RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions
par: Zhang, Yuansen, et autres
Publié: (2024)

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
par: Wu, Haoyuan, et autres
Publié: (2025)

D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
par: Wang, Haodong, et autres
Publié: (2025)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
par: Zhou, Yuhao, et autres
Publié: (2025)

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
par: Ye, Junjie, et autres
Publié: (2024)

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning
par: Yang, Shu, et autres
Publié: (2024)

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
par: Xia, Han, et autres
Publié: (2024)

LLaDA-MoE: A Sparse MoE Diffusion Language Model
par: Zhu, Fengqi, et autres
Publié: (2025)

LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
par: Wang, Shaoheng, et autres
Publié: (2025)

MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems
par: Zhou, Zhuoshan, et autres
Publié: (2026)

Hierarchical LoRA MoE for Efficient CTR Model Scaling
par: Zeng, Zhichen, et autres
Publié: (2025)

Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
par: Prottasha, Nusrat Jahan, et autres
Publié: (2026)

ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
par: Xu, Yuhao, et autres
Publié: (2026)

Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
par: Wang, Junzhe, et autres
Publié: (2026)

MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
par: Su, Zhaoyuan, et autres
Publié: (2026)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
par: Hannah, Lauren. A, et autres
Publié: (2025)

Secrets of RLHF in Large Language Models Part II: Reward Modeling
par: Wang, Binghai, et autres
Publié: (2024)

MetaRM: Shifted Distributions Alignment via Meta-Learning
par: Dou, Shihan, et autres
Publié: (2024)

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
par: Cao, Shiyi, et autres
Publié: (2024)

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
par: Qian, Yulei, et autres
Publié: (2024)

GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
par: Wu, Haoze, et autres
Publié: (2024)

Pre-Trained Policy Discriminators are General Reward Models
par: Dou, Shihan, et autres
Publié: (2025)

Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
par: Tang, Yehui, et autres
Publié: (2025)