Image de couverture de livre

The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lyu, Xingyu, Xu, Qianqian, Yang, Zhiyong, Wen, Peisong, Huang, Qingming
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
par: Lei, Qifeng, et autres
Publié: (2025)

GLU Attention Improve Transformer
par: Wang, Zehao
Publié: (2025)

Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models
par: Guo, Zhiyu, et autres
Publié: (2024)

AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation
par: Han, Boyu, et autres
Publié: (2024)

PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks
par: Medeiros, Daniel Nobrega
Publié: (2026)

Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
par: Liu, Ziyang
Publié: (2026)

Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
par: Lau, Tim Tsz-Kit, et autres
Publié: (2026)

Optimizing Partial Area Under the Top-k Curve: Theory and Practice
par: Wang, Zitai, et autres
Publié: (2022)

SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAM
par: Lyu, Xingyu, et autres
Publié: (2024)

ReconBoost: Boosting Can Achieve Modality Reconcilement
par: Hua, Cong, et autres
Publié: (2024)

Two Heads Are Better than One: Model-Weight and Latent-Space Analysis for Federated Learning on Non-iid Data against Poisoning Attacks
par: Lyu, Xingyu, et autres
Publié: (2025)

Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection
par: Han, Boyu, et autres
Publié: (2026)

LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
par: Han, Boyu, et autres
Publié: (2025)

DirMixE: Harnessing Test Agnostic Long-tail Recognition with Hierarchical Label Vartiations
par: Yang, Zhiyong, et autres
Publié: (2024)

Dual-Stage Reweighted MoE for Long-Tailed Egocentric Mistake Detection
par: Han, Boyu, et autres
Publié: (2025)

A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization
par: Wang, Zitai, et autres
Publié: (2023)

Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs
par: Yang, Jaewoo, et autres
Publié: (2024)

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation
par: Han, Boyu, et autres
Publié: (2026)

Towards Size-invariant Salient Object Detection: A Generic Evaluation and Optimization Approach
par: Bao, Shilong, et autres
Publié: (2025)

HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution Detection
par: He, Junwei, et autres
Publié: (2024)

GLU: Global-Local-Uncertainty Fusion for Scalable Spatiotemporal Reconstruction and Forecasting
par: Wang, Linzheng, et autres
Publié: (2026)

GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models
par: Gerstner, Sebastian, et autres
Publié: (2026)

Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification
par: Wang, Zitai, et autres
Publié: (2024)

AudioMAE++: learning better masked audio representations with SwiGLU FFNs
par: Yadav, Sarthak, et autres
Publié: (2025)

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
par: Zhao, Ziyu, et autres
Publié: (2026)

Semantic Concentration for Self-Supervised Dense Representations Learning
par: Wen, Peisong, et autres
Publié: (2025)

FiCoTS: Fine-to-Coarse LLM-Enhanced Hierarchical Cross-Modality Interaction for Time Series Forecasting
par: Lyu, Yafei, et autres
Publié: (2025)

Is Data Shapley Not Better than Random in Data Selection? Ask NASH
par: Tian, Xiao, et autres
Publié: (2026)

Angel or Devil: Discriminating Hard Samples and Anomaly Contaminations for Unsupervised Time Series Anomaly Detection
par: Zhang, Ruyi, et autres
Publié: (2024)

Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques
par: Meng, Benyuan, et autres
Publié: (2024)

Two Heads Are Better than One: Simulating Large Transformers with Small Ones
par: Yu, Hantao, et autres
Publié: (2025)

Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
par: Zhao, Wanru, et autres
Publié: (2026)

This Looks Better than That: Better Interpretable Models with ProtoPNeXt
par: Willard, Frank, et autres
Publié: (2024)

Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
par: Choudhury, Sanjiban, et autres
Publié: (2024)

SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
par: Piras, Giorgio, et autres
Publié: (2025)

EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
par: Chen, Yuanteng, et autres
Publié: (2025)

HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
par: Xie, Zhinan, et autres
Publié: (2025)

Ban&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing
par: Chen, Yuanteng, et autres
Publié: (2025)

The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
par: Wang, Guanghui, et autres
Publié: (2026)

Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
par: Chen, Xi, et autres
Publié: (2026)