Imagen de Portada

Cutting the Skip: Training Residual-Free Transformers

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Ji, Yiping, Martens, James, Zheng, Jianqiao, Zhou, Ziqin, Moghadam, Peyman, Zhang, Xinyu, Saratchandran, Hemanth, Lucey, Simon
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Always Skip Attention
por: Ji, Yiping, et al.
Publicado: (2025)

The Quantization Benefits of Residual-Free Transformers
por: Ji, Yiping, et al.
Publicado: (2026)

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy
por: Zheng, Jianqiao, et al.
Publicado: (2026)

From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
por: Zheng, Jianqiao, et al.
Publicado: (2025)

Spectral Conditioning of Attention Improves Transformer Performance
por: Saratchandran, Hemanth, et al.
Publicado: (2026)

Leaner Transformers: More Heads, Less Depth
por: Saratchandran, Hemanth, et al.
Publicado: (2025)

SineLoRA$Δ$: Sine-Activated Delta Compression
por: Gordon, Cameron, et al.
Publicado: (2025)

Preconditioners for the Stochastic Training of Neural Fields
por: Chng, Shin-Fang, et al.
Publicado: (2024)

Efficient Learning With Sine-Activated Low-rank Matrices
por: Ji, Yiping, et al.
Publicado: (2024)

From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Architectural Strategies for the optimization of Physics-Informed Neural Networks
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Preconditioned Attention: Enhancing Efficiency in Transformers
por: Saratchandran, Hemanth
Publicado: (2026)

Structured Initialization for Vision Transformers
por: Zheng, Jianqiao, et al.
Publicado: (2025)

Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
por: Garg, Arpit, et al.
Publicado: (2025)

Enhancing Transformers Through Conditioned Embedded Tokens
por: Saratchandran, Hemanth, et al.
Publicado: (2025)

A Sampling Theory Perspective on Activations for Implicit Neural Representations
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations
por: Gordon, Cameron, et al.
Publicado: (2024)

SineProject: Machine Unlearning for Stable Vision Language Alignment
por: Garg, Arpit, et al.
Publicado: (2025)

Weight Conditioning for Smooth Optimization of Neural Networks
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
por: Shinnick, Zachary, et al.
Publicado: (2025)

Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
por: Yao, Jiaqi, et al.
Publicado: (2025)

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
por: Xu, Runze, et al.
Publicado: (2026)

Invertible Neural Warp for NeRF
por: Chng, Shin-Fang, et al.
Publicado: (2024)

Gradient Descent as a Shrinkage Operator for Spectral Bias
por: Lucey, Simon
Publicado: (2025)

Spatioformer: A Geo-encoded Transformer for Large-Scale Plant Species Richness Prediction
por: Guo, Yiqing, et al.
Publicado: (2024)

Structured Initialization for Attention in Vision Transformers
por: Zheng, Jianqiao, et al.
Publicado: (2024)

Convolutional Initialization for Data-Efficient Vision Transformers
por: Zheng, Jianqiao, et al.
Publicado: (2024)

Procedural Pretraining: Warming Up Language Models with Abstract Data
por: Jiang, Liangze, et al.
Publicado: (2026)

Flashbacks to Harmonize Stability and Plasticity in Continual Learning
por: Mahmoodi, Leila, et al.
Publicado: (2025)

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
por: Albert, Paul, et al.
Publicado: (2025)

Inductive Graph Few-shot Class Incremental Learning
por: Li, Yayong, et al.
Publicado: (2024)

ReadMOF: Structure-Free Semantic Embeddings from Systematic MOF Nomenclature for Machine Learning
por: Zhu, Kewei, et al.
Publicado: (2026)

Learning to Skip the Middle Layers of Transformers
por: Lawson, Tim, et al.
Publicado: (2025)

GPT Carry-On: Training Foundation Model for Customization Could Be Simple, Scalable and Affordable
por: Wangni, Jianqiao
Publicado: (2025)

Enhancing Linear Attention with Residual Learning
por: Lai, Xunhao, et al.
Publicado: (2025)

Large-Scale In-Game Outcome Forecasting for Match, Team and Players in Football using an Axial Transformer Neural Network
por: Horton, Michael, et al.
Publicado: (2025)

Plant species richness prediction from DESIS hyperspectral data: A comparison study on feature extraction procedures and regression models
por: Guo, Yiqing, et al.
Publicado: (2023)

A Class of Dependent Random Distributions Based on Atom Skipping
por: Bi, Dehua, et al.
Publicado: (2023)