How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers

Documenti analoghi

• How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
  di: Lu, Xin, et al.
  Pubblicazione: (2025)
• Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
  di: Pei, Zehua, et al.
  Pubblicazione: (2025)
• Continual Pre-training of MoEs: How robust is your router?
  di: Thérien, Benjamin, et al.
  Pubblicazione: (2025)
• SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
  di: Tang, Shengkun, et al.
  Pubblicazione: (2026)
• MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usability
  di: Du, Yanrui, et al.
  Pubblicazione: (2024)