Cover Image

Leaner Transformers: More Heads, Less Depth

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Saratchandran, Hemanth, Teney, Damien, Lucey, Simon
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
by: Saratchandran, Hemanth, et al.
Published: (2024)

Enhancing Transformers Through Conditioned Embedded Tokens
by: Saratchandran, Hemanth, et al.
Published: (2025)

Preconditioners for the Stochastic Training of Neural Fields
by: Chng, Shin-Fang, et al.
Published: (2024)

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
by: Saratchandran, Hemanth, et al.
Published: (2024)

Always Skip Attention
by: Ji, Yiping, et al.
Published: (2025)

From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
by: Zheng, Jianqiao, et al.
Published: (2025)

D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations
by: Gordon, Cameron, et al.
Published: (2024)

Efficient Learning With Sine-Activated Low-rank Matrices
by: Ji, Yiping, et al.
Published: (2024)

Structured Initialization for Vision Transformers
by: Zheng, Jianqiao, et al.
Published: (2025)

SineProject: Machine Unlearning for Stable Vision Language Alignment
by: Garg, Arpit, et al.
Published: (2025)

Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
by: Shinnick, Zachary, et al.
Published: (2025)

Weight Conditioning for Smooth Optimization of Neural Networks
by: Saratchandran, Hemanth, et al.
Published: (2024)

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
by: Xu, Runze, et al.
Published: (2026)

Invertible Neural Warp for NeRF
by: Chng, Shin-Fang, et al.
Published: (2024)

Spectral Conditioning of Attention Improves Transformer Performance
by: Saratchandran, Hemanth, et al.
Published: (2026)

Gradient Descent as a Shrinkage Operator for Spectral Bias
by: Lucey, Simon
Published: (2025)

Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
by: Teney, Damien, et al.
Published: (2025)

Neural Redshift: Random Networks are not Random Functions
by: Teney, Damien, et al.
Published: (2024)

Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
by: Liu, Zheyuan, et al.
Published: (2023)

Scalable Ensemble Diversification for OOD Generalization and Detection
by: Rubinstein, Alexander, et al.
Published: (2024)

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
by: Albert, Paul, et al.
Published: (2025)

Less-to-More Generalization: Unlocking More Controllability by In-Context Generation
by: Wu, Shaojin, et al.
Published: (2025)

Rethinking the Role of Spatial Mixing
by: Cazenavette, George, et al.
Published: (2025)

Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
by: Bochkovskii, Aleksei, et al.
Published: (2024)

Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles
by: Scimeca, Luca, et al.
Published: (2023)

Less is More: Fewer Interpretable Region via Submodular Subset Selection
by: Chen, Ruoyu, et al.
Published: (2024)

Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets
by: Pereg, Deborah, et al.
Published: (2022)

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy
by: Zheng, Jianqiao, et al.
Published: (2026)

Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection
by: Chen, Ruoyu, et al.
Published: (2025)

Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning
by: Alssum, Lama, et al.
Published: (2025)

Continual Learning: Less Forgetting, More OOD Generalization via Adaptive Contrastive Replay
by: Rezaei, Hossein, et al.
Published: (2024)

From Channel Bias to Feature Redundancy: Uncovering the "Less is More" Principle in Few-Shot Learning
by: Zhang, Ji, et al.
Published: (2023)

3D-LFM: Lifting Foundation Model
by: Dabhi, Mosam, et al.
Published: (2023)

Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
by: Rodriguez-Opazo, Cristian, et al.
Published: (2024)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition
by: Rahimi, Parsa, et al.
Published: (2025)

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
by: Shinnick, Zachary, et al.
Published: (2025)

Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
by: Saratchandran, Hemanth, et al.
Published: (2024)

Architectural Strategies for the optimization of Physics-Informed Neural Networks
by: Saratchandran, Hemanth, et al.
Published: (2024)

AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
by: Jajal, Purvish, et al.
Published: (2025)