Cover Image

Residual Matrix Transformers: Scaling the Size of the Residual Stream

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mak, Brian, Flanigan, Jeffrey
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Residual Stream Duality in Modern Transformer Architectures
by: Zhang, Yifan
Published: (2026)

Residual Stream Analysis with Multi-Layer SAEs
by: Lawson, Tim, et al.
Published: (2024)

ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
by: Wang, Yiming, et al.
Published: (2023)

RAC: Efficient LLM Factuality Correction with Retrieval Augmentation
by: Li, Changmao, et al.
Published: (2024)

Residual-Mass Accounting for Partial-KV Decoding
by: Hoshi, Yasuto, et al.
Published: (2026)

Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams
by: Llorente-Saguer, Isaac
Published: (2026)

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
by: Xiao, Da, et al.
Published: (2025)

Large Language Model Unlearning via Embedding-Corrupted Prompts
by: Liu, Chris Yuhao, et al.
Published: (2024)

Jet Expansions of Residual Computation
by: Chen, Yihong, et al.
Published: (2024)

M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference
by: Bhendawade, Nikhil, et al.
Published: (2025)

Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering
by: Gupta, Manan, et al.
Published: (2026)

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training
by: Li, Qingyang, et al.
Published: (2024)

Neutral Residues: Revisiting Adapters for Model Extension
by: Talla, Franck Signe, et al.
Published: (2024)

SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
by: Joshi, Maithili, et al.
Published: (2025)

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
by: Zhou, Wuyang, et al.
Published: (2026)

Residual Speech Embeddings for Tone Classification: Removing Linguistic Content to Enhance Paralinguistic Analysis
by: Ahbabi, Hamdan Al, et al.
Published: (2025)

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning
by: Xia, Wenhan, et al.
Published: (2024)

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
by: Saxena, Utkarsh, et al.
Published: (2024)

S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
by: Zeng, Hanqing, et al.
Published: (2025)

ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning
by: Lin, Zihan, et al.
Published: (2026)

The Geometry of Harmful Intent: Training-Free Anomaly Detection via Angular Deviation in LLM Residual Streams
by: Llorente-Saguer, Isaac
Published: (2026)

SR-TTT: Surprisal-Aware Residual Test-Time Training
by: P, Swamynathan V
Published: (2026)

Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing
by: Jain, Manish, et al.
Published: (2025)

QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering
by: Wang, Yanshu, et al.
Published: (2024)

Naturally Computed Scale Invariance in the Residual Stream of ResNet18
by: Longon, André
Published: (2025)

Residual-based Language Models are Free Boosters for Biomedical Imaging
by: Lai, Zhixin, et al.
Published: (2024)

Hierarchical Orthogonal Residual Spread for Precise Massive Editing in Large Language Models
by: Gu, Xiaojie, et al.
Published: (2026)

Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
by: Yuan, Yurun, et al.
Published: (2025)

Interpreting the Residual Stream of ResNet18
by: Longon, André
Published: (2024)

Scaling Law for Language Models Training Considering Batch Size
by: Shuai, Xian, et al.
Published: (2024)

Towards Robust FastSpeech 2 by Modelling Residual Multimodality
by: Kögel, Fabian, et al.
Published: (2023)

Momentum Streams for Optimizer-Inspired Transformers
by: Gai, Jingchu, et al.
Published: (2026)

Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization
by: Wang, YiFeng, et al.
Published: (2026)

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
by: Brandon, William, et al.
Published: (2024)

State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
by: Aviss, Thea
Published: (2026)

StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data
by: Pellegrain, Victor, et al.
Published: (2021)

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
by: Huang, Hongzhi, et al.
Published: (2025)

Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes
by: Liu, Ming
Published: (2026)

The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
by: Kerce, J. Clayton, et al.
Published: (2026)

Toward Preference-aligned Large Language Models via Residual-based Model Steering
by: La Cava, Lucio, et al.
Published: (2025)