Imagen de Portada

LayerNorm Induces Recency Bias in Transformer Decoders

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Kim, Junu, Liu, Xiao, Lin, Zhenghao, Ji, Lei, Gong, Yeyun, Choi, Edward
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

SLaNC: Static LayerNorm Calibration
por: Salmani, Mahsa, et al.
Publicado: (2024)

Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
por: Chen, Chen, et al.
Publicado: (2026)

You can remove GPT2's LayerNorm by fine-tuning
por: Heimersheim, Stefan
Publicado: (2024)

LayerNorm: A key component in parameter-efficient fine-tuning
por: ValizadehAslani, Taha, et al.
Publicado: (2024)

When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
por: Verma, Lucky
Publicado: (2026)

On the Role of Attention Masks and LayerNorm in Transformers
por: Wu, Xinyi, et al.
Publicado: (2024)

Geometry and Dynamics of LayerNorm
por: Riechers, Paul M.
Publicado: (2024)

Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
por: Baroni, Luca, et al.
Publicado: (2025)

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
por: Shin, Haebin, et al.
Publicado: (2025)

Impact of Layer Norm on Memorization and Generalization in Transformers
por: Singhal, Rishi, et al.
Publicado: (2025)

SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
por: Wang, Wenxun, et al.
Publicado: (2025)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
por: Yang, Kailai, et al.
Publicado: (2025)

General-Purpose Retrieval-Enhanced Medical Prediction Model Using Near-Infinite History
por: Kim, Junu, et al.
Publicado: (2023)

MoETTA: Test-Time Adaptation Under Mixed Distribution Shifts with MoE-LayerNorm
por: Fan, Xiao, et al.
Publicado: (2025)

APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning
por: Sun, Jiashuo, et al.
Publicado: (2022)

Integrative Decoding: Improve Factuality via Implicit Self-consistency
por: Cheng, Yi, et al.
Publicado: (2024)

Optimizing Large Language Model Training Using FP4 Quantization
por: Wang, Ruizhe, et al.
Publicado: (2025)

Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning
por: Park, Sungjin, et al.
Publicado: (2024)

Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification
por: Qian, Wenhao, et al.
Publicado: (2025)

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
por: Bae, Sangmin, et al.
Publicado: (2024)

DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding
por: Zarch, Hossein Entezari, et al.
Publicado: (2025)

GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
por: Zheng, Chuanyang, et al.
Publicado: (2026)

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
por: Wang, Chao, et al.
Publicado: (2026)

PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration
por: Wu, Songhao, et al.
Publicado: (2025)

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
por: Liang, Xiao, et al.
Publicado: (2025)

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
por: Amer, Walaa, et al.
Publicado: (2026)

On The Adaptation of Unlimiformer for Decoder-Only Transformers
por: Ahrabian, Kian, et al.
Publicado: (2024)

Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation
por: Gui, Lujun, et al.
Publicado: (2024)

Improving Data and Reward Design for Scientific Reasoning in Large Language Models
por: Chen, Zijie, et al.
Publicado: (2026)

Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
por: Shin, Haebin, et al.
Publicado: (2025)

Red-Teaming for Inducing Societal Bias in Large Language Models
por: Luo, Chu Fei, et al.
Publicado: (2024)

Federated Learning for Heterogeneous Electronic Health Record Systems with Cost Effective Participant Selection
por: Kim, Jiyoun, et al.
Publicado: (2024)

Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
por: Sharma, Aryan, et al.
Publicado: (2026)

How Powerful are Decoder-Only Transformer Neural Models?
por: Roberts, Jesse
Publicado: (2023)

Learning to Skip the Middle Layers of Transformers
por: Lawson, Tim, et al.
Publicado: (2025)

DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
por: Chuang, Yung-Sung, et al.
Publicado: (2023)

Linear Recency Bias During Training Improves Transformers' Fit to Reading Times
por: Clark, Christian, et al.
Publicado: (2024)

Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
por: Xiao, Liu
Publicado: (2026)

Faster Transformer Decoding: N-gram Masked Self-Attention
por: Chelba, Ciprian, et al.
Publicado: (2020)

Multi-lingual Multi-institutional Electronic Health Record based Predictive Model
por: Hur, Kyunghoon, et al.
Publicado: (2026)