Imagen de Portada

Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Li, Hongkang, Wang, Meng, Zhang, Shuai, Liu, Sijia, Chen, Pin-Yu
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers
por: Li, Hongkang, et al.
Publicado: (2025)

How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance
por: Li, Hongkang, et al.
Publicado: (2024)

What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
por: Li, Hongkang, et al.
Publicado: (2024)

How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?
por: Li, Hongkang, et al.
Publicado: (2024)

Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis
por: Li, Hongkang, et al.
Publicado: (2024)

Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis
por: Li, Hongkang, et al.
Publicado: (2025)

Visual prompting reimagined: The power of the Activation Prompts
por: Zhang, Yihua, et al.
Publicado: (2026)

SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
por: Zhang, Shuai, et al.
Publicado: (2024)

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
por: Chowdhury, Mohammed Nowaz Rabbani, et al.
Publicado: (2024)

A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
por: Shandirasegaran, Mugunthan, et al.
Publicado: (2026)

Provable Meta-Learning with Low-Rank Adaptations
por: Block, Jacob L., et al.
Publicado: (2024)

Low-Rank Tensor Learning by Generalized Nonconvex Regularization
por: Xia, Sijia, et al.
Publicado: (2024)

Transformers Provably Learn Sparse XOR with Polylogarithmic Parameters
por: Han, Yaomengxi, et al.
Publicado: (2025)

Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
por: Sun, Jiawei, et al.
Publicado: (2025)

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
por: Li, Zihao, et al.
Publicado: (2024)

Provable Exactness for Asymmetric Low-Rank SDP Learning
por: Hu, Enliang
Publicado: (2018)

Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs
por: Li, Hongkang, et al.
Publicado: (2026)

Provable Multi-Task Reinforcement Learning: A Representation Learning Framework with Low Rank Rewards
por: Guo, Yaoze, et al.
Publicado: (2026)

Provable Knowledge Acquisition and Extraction in One-Layer Transformers
por: Xu, Ruichen, et al.
Publicado: (2025)

Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
por: Shen, Li, et al.
Publicado: (2026)

Low-Rank Adapting Models for Sparse Autoencoders
por: Chen, Matthew, et al.
Publicado: (2025)

PSBD: Prediction Shift Uncertainty Unlocks Backdoor Detection
por: Li, Wei, et al.
Publicado: (2024)

Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
por: Huang, Ruiquan, et al.
Publicado: (2026)

Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot
por: Wang, Zixuan, et al.
Publicado: (2024)

Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions
por: Chai, Jinhang, et al.
Publicado: (2026)

Provably Learning from Modern Language Models via Low Logit Rank
por: Golowich, Noah, et al.
Publicado: (2025)

Learning Cascade Ranking as One Network
por: Wang, Yunli, et al.
Publicado: (2025)

Provable Training for Graph Contrastive Learning
por: Yu, Yue, et al.
Publicado: (2023)

Dynamic Low-Rank Sparse Adaptation for Large Language Models
por: Huang, Weizhong, et al.
Publicado: (2025)

3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
por: Makni, Mehdi, et al.
Publicado: (2026)

Provable In-Context Learning of Nonlinear Regression with Transformers
por: Li, Hongbo, et al.
Publicado: (2025)

InRank: Incremental Low-Rank Learning
por: Zhao, Jiawei, et al.
Publicado: (2023)

Transformers Provably Learn to Internalize Chain-of-Thought
por: Huang, Yixiao, et al.
Publicado: (2026)

Transformer Learns Optimal Variable Selection in Group-Sparse Classification
por: Zhang, Chenyang, et al.
Publicado: (2025)

Provably learning a multi-head attention layer
por: Chen, Sitan, et al.
Publicado: (2024)

Unlabeled Data Can Provably Enhance In-Context Learning of Transformers
por: Liu, Renpu, et al.
Publicado: (2026)

Consensus Knowledge Graph Learning via Multi-view Sparse Low Rank Block Model
por: Cai, Tianxi, et al.
Publicado: (2022)

SARA: Singular-Value Based Adaptive Low-Rank Adaption
por: Gu, Jihao, et al.
Publicado: (2024)

Provably Efficient Online RLHF with One-Pass Reward Modeling
por: Li, Long-Fei, et al.
Publicado: (2025)

LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
por: Wang, Changsheng, et al.
Publicado: (2025)