More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Documenti analoghi

• Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
  di: Wang, Mingze, et al.
  Pubblicazione: (2026)
• More Expressive Attention with Negative Weights
  di: Lv, Ang, et al.
  Pubblicazione: (2024)
• DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
  di: Gülmez, Gökdeniz
  Pubblicazione: (2026)
• On the Expressive Power and Limitations of Multi-Layer SSMs
  di: Zubić, Nikola, et al.
  Pubblicazione: (2026)
• The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
  di: Wang, Jinbo, et al.
  Pubblicazione: (2025)