Cover Image

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Mingze, Wang, Jinbo, Xia, Yikuan, Shen, Kai, Zhong, Shu
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
by: Wang, Mingze, et al.
Published: (2026)

More Expressive Attention with Negative Weights
by: Lv, Ang, et al.
Published: (2024)

DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
by: Gülmez, Gökdeniz
Published: (2026)

On the Expressive Power and Limitations of Multi-Layer SSMs
by: Zubić, Nikola, et al.
Published: (2026)

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
by: Wang, Jinbo, et al.
Published: (2025)

Learning More Expressive General Policies for Classical Planning Domains
by: Ståhlberg, Simon, et al.
Published: (2024)

Global-Lens Transformers: Adaptive Token Mixing for Dynamic Link Prediction
by: Zou, Tao, et al.
Published: (2025)

LARV: Data-Free Layer-wise Adaptive Rescaling Veneer for Model Merging
by: Wang, Xinyu, et al.
Published: (2026)

Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
by: Taniguchi, Rei, et al.
Published: (2026)

Will More Expressive Graph Neural Networks do Better on Generative Tasks?
by: Zou, Xiandong, et al.
Published: (2023)

ANAct: Adaptive Normalization for Activation Functions
by: Peiwen, Yuan, et al.
Published: (2022)

Deep Reinforcement Learning for Efficient and Fair Allocation of Health Care Resources
by: Li, Yikuan, et al.
Published: (2023)

An Empirical Study of Realized GNN Expressiveness
by: Wang, Yanbo, et al.
Published: (2023)

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
by: Wang, Keyu, et al.
Published: (2025)

Are More Tokens Rational? Inference-Time Scaling in Language Models as Adaptive Resource Rationality
by: Hu, Zhimin, et al.
Published: (2026)

ToMA: Token Merge with Attention for Diffusion Models
by: Lu, Wenbo, et al.
Published: (2025)

SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks
by: Omidvar, Amin
Published: (2025)

AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models
by: Wang, Xiangqi, et al.
Published: (2025)

Composing Parts for Expressive Object Generation
by: Rangwani, Harsh, et al.
Published: (2024)

Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency
by: Jiang, Xinyan, et al.
Published: (2026)

Are Expressive Models Truly Necessary for Offline RL?
by: Wang, Guan, et al.
Published: (2024)

Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
by: Armstrong, Marcus, et al.
Published: (2026)

Compression Repair for Feedforward Neural Networks Based on Model Equivalence Evaluation
by: Mo, Zihao, et al.
Published: (2024)

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
by: Chen, Lei, et al.
Published: (2026)

VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
by: Zhou, Jingbo, et al.
Published: (2026)

Deriving Equivalent Symbol-Based Decision Models from Feedforward Neural Networks
by: Seidel, Sebastian, et al.
Published: (2025)

Adaptive Token-Weighted Differential Privacy for LLMs: Not All Tokens Require Equal Protection
by: Yu, Manjiang, et al.
Published: (2025)

Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
by: Zhang, Zhi, et al.
Published: (2026)

PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization
by: Wang, Xinhai, et al.
Published: (2025)

CombAlign: Enhancing Model Expressiveness in Unsupervised Graph Alignment
by: Chen, Songyang, et al.
Published: (2024)

Token Activation Map to Visually Explain Multimodal LLMs
by: Li, Yi, et al.
Published: (2025)

AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuning
by: Song, Yurun, et al.
Published: (2025)

MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks
by: Hong, Zhi, et al.
Published: (2026)

MetaSymNet: A Tree-like Symbol Network with Adaptive Architecture and Activation Functions
by: Li, Yanjie, et al.
Published: (2023)

One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
by: Lu, Liming, et al.
Published: (2026)

Stable but Wrong: When More Data Degrades Scientific Conclusions
by: Zhang, Zhipeng, et al.
Published: (2026)

Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
by: Zhao, Lifan, et al.
Published: (2025)

LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization
by: Wang, Boxiao, et al.
Published: (2026)

Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
by: Yun, Vincent-Daniel, et al.
Published: (2026)

DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models
by: Shen, Yi, et al.
Published: (2025)