Cover Image

Gating is Weighting: Understanding Gated Linear Attention through In-context Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Yingcong, Tarzanagh, Davoud Ataee, Rawat, Ankit Singh, Fazel, Maryam, Oymak, Samet
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Transformers as Support Vector Machines
by: Tarzanagh, Davoud Ataee, et al.
Published: (2023)

Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
by: Li, Yingcong, et al.
Published: (2024)

Mechanics of Next Token Prediction with Self-Attention
by: Li, Yingcong, et al.
Published: (2024)

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds
by: Sattar, Yahya, et al.
Published: (2021)

When and How Unlabeled Data Provably Improve In-Context Learning
by: Li, Yingcong, et al.
Published: (2025)

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
by: Ildiz, M. Emrullah, et al.
Published: (2024)

Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection
by: Julistiono, Addison Kristanto, et al.
Published: (2024)

A Penalty-Based Method for Communication-Efficient Decentralized Bilevel Programming
by: Nazari, Parvin, et al.
Published: (2022)

Online Bilevel Optimization: Regret Analysis of Online Alternating Gradient Methods
by: Tarzanagh, Davoud Ataee, et al.
Published: (2022)

Stochastic Regret Guarantees for Online Zeroth- and First-Order Bilevel Optimization
by: Nazari, Parvin, et al.
Published: (2025)

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
by: Farhat, Yehya, et al.
Published: (2023)

Provable Benefits of Task-Specific Prompts for In-context Learning
by: Chang, Xiangyu, et al.
Published: (2025)

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective
by: Fernando, Heshan, et al.
Published: (2024)

Truncated Matrix Completion - An Empirical Study
by: Naik, Rishhabh, et al.
Published: (2025)

Temporal Robustness in Discrete Time Linear Dynamical Systems
by: Metya, Nilava, et al.
Published: (2025)

Can Transformers Learn Optimal Filtering for Unknown Systems?
by: Balim, Haldun, et al.
Published: (2023)

Variational Learning is Effective for Large Deep Networks
by: Shen, Yuesong, et al.
Published: (2024)

Reinforcement Learning from Human Feedback with Active Queries
by: Ji, Kaixuan, et al.
Published: (2024)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
by: Chen, Siyu, et al.
Published: (2024)

ACING: Actor-Critic for Instruction Learning in Black-Box LLMs
by: Kharrat, Salma, et al.
Published: (2024)

Learning Backdoors for Mixed Integer Linear Programs with Contrastive Learning
by: Cai, Junyang, et al.
Published: (2024)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
by: Meterez, Alexandru, et al.
Published: (2026)

Understanding Optimization in Deep Learning with Central Flows
by: Cohen, Jeremy M., et al.
Published: (2024)

Multi-task Representation Learning for Mixed Integer Linear Programming
by: Cai, Junyang, et al.
Published: (2024)

Explore-then-Commit for Nonstationary Linear Bandits with Latent Dynamics
by: Choi, Sunmook, et al.
Published: (2025)

How Well Can Transformers Emulate In-context Newton's Method?
by: Giannou, Angeliki, et al.
Published: (2024)

Optimization and Generalization Guarantees for Weight Normalization
by: Cisneros-Velarde, Pedro, et al.
Published: (2024)

Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
by: Su, Weijie
Published: (2025)

Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems
by: Huang, Yilie, et al.
Published: (2024)

Solving General Natural-Language-Description Optimization Problems with Large Language Models
by: Zhang, Jihai, et al.
Published: (2024)

One-Shot Safety Alignment for Large Language Models via Optimal Dualization
by: Huang, Xinmeng, et al.
Published: (2024)

Causal LLM Routing: End-to-End Regret Minimization from Observational Data
by: Tsiourvas, Asterios, et al.
Published: (2025)

Reward Collapse in Aligning Large Language Models
by: Song, Ziang, et al.
Published: (2023)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
by: Gautam, Tanmay, et al.
Published: (2024)

DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
by: Gurses, Selcuk, et al.
Published: (2025)

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
by: Pan, Rui, et al.
Published: (2024)

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
by: Tao, Hongyi, et al.
Published: (2026)

Leveraging Large Language Models for Solving Rare MIP Challenges
by: Wang, Teng, et al.
Published: (2024)

Unsupervised Machine Learning Hybrid Approach Integrating Linear Programming in Loss Function: A Robust Optimization Technique
by: Kiruluta, Andrew, et al.
Published: (2024)

ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertise
by: Guo, Xingang, et al.
Published: (2024)