Cover Image

An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Yunzhe, Zou, Difan, Xu, Dong
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization
by: Hu, Yunzhe, et al.
Published: (2025)

How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression
by: Chen, Xingwu, et al.
Published: (2024)

Model Unlearning via Sparse Autoencoder Subspace Guided Projections
by: Wang, Xu, et al.
Published: (2025)

What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks
by: Chen, Xingwu, et al.
Published: (2024)

Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders
by: Wang, Xu, et al.
Published: (2025)

Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression
by: Chen, Xingwu, et al.
Published: (2025)

On the Memorization of Consistency Distillation for Diffusion Models
by: Jiang, Bingqing, et al.
Published: (2026)

Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference
by: Han, Yujin, et al.
Published: (2024)

DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders
by: Wang, Xu, et al.
Published: (2026)

A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models
by: Xie, Chengxing, et al.
Published: (2024)

Structured Role-Aware Policy Optimization for Multimodal Reasoning
by: Jiang, Bingqing, et al.
Published: (2026)

On the Feature Learning in Diffusion Models
by: Han, Andi, et al.
Published: (2024)

Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generation
by: Zhang, Yi, et al.
Published: (2025)

The Implicit Bias of Adam on Separable Data
by: Zhang, Chenyang, et al.
Published: (2024)

Improving Implicit Regularization of SGD with Preconditioning for Least Square Problems
by: Su, Junwei, et al.
Published: (2024)

PRES: Toward Scalable Memory-Based Dynamic Graph Neural Networks
by: Su, Junwei, et al.
Published: (2024)

Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
by: Bai, Hanru, et al.
Published: (2025)

Physics-Informed Neural PDE Solvers via Spatio-Temporal MeanFlow
by: Bai, Hanru, et al.
Published: (2026)

On the Limitation and Experience Replay for GNNs in Continual Learning
by: Su, Junwei, et al.
Published: (2023)

The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient
by: Li, Jichu, et al.
Published: (2026)

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
by: Yu, Yaodong, et al.
Published: (2023)

Faster Sampling via Stochastic Gradient Proximal Sampler
by: Huang, Xunpeng, et al.
Published: (2024)

On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
by: Zhang, Yi, et al.
Published: (2025)

An Improved Analysis of Langevin Algorithms with Prior Diffusion for Non-Log-Concave Sampling
by: Huang, Xunpeng, et al.
Published: (2024)

Learning under Quantization for High-Dimensional Linear Regression
by: Zhang, Dechen, et al.
Published: (2025)

Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
by: Chen, Xingwu, et al.
Published: (2025)

A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization
by: Tang, Xuan, et al.
Published: (2025)

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
by: Gao, Peifeng, et al.
Published: (2026)

Capturing Conditional Dependence via Auto-regressive Diffusion Models
by: Huang, Xunpeng, et al.
Published: (2025)

On the Robustness of Transformers against Context Hijacking for Linear Classification
by: Li, Tianle, et al.
Published: (2025)

Towards Robust Graph Incremental Learning on Evolving Graphs
by: Su, Junwei, et al.
Published: (2024)

On the Benefits of Over-parameterization for Out-of-Distribution Generalization
by: Hao, Yifan, et al.
Published: (2024)

Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers
by: Shang, Shuning, et al.
Published: (2024)

F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
by: Zhang, Hangwei, et al.
Published: (2025)

Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks
by: Zhang, Chenyang, et al.
Published: (2025)

Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference
by: Huang, Xunpeng, et al.
Published: (2024)

Targeted Maximum Likelihood Estimation for Integral Projection Models in Population Ecology
by: Zhou, Yunzhe, et al.
Published: (2024)

Faster Sampling without Isoperimetry via Diffusion-based Monte Carlo
by: Huang, Xunpeng, et al.
Published: (2024)

Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction
by: Wu, Ziyang, et al.
Published: (2024)

Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis
by: Wang, Xu, et al.
Published: (2025)