Cover Image

TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Menezes, Michael, Su, Barbara, Feng, Xinze, Farhat, Yehya, Shili, Hamza, Kyrillidis, Anastasios
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
by: Farhat, Yehya, et al.
Published: (2023)

One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
by: Shen, Jucheng, et al.
Published: (2026)

On the Robustness of Decision-Focused Learning
by: Farhat, Yehya
Published: (2023)

Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
by: Liao, Fangshuo, et al.
Published: (2025)

Provable Accelerated Convergence of Nesterov's Momentum for Deep ReLU Neural Networks
by: Liao, Fangshuo, et al.
Published: (2023)

Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation
by: Liao, Fangshuo, et al.
Published: (2025)

SGD at the Edge of Stability: The Stochastic Sharpness Gap
by: Liao, Fangshuo, et al.
Published: (2026)

When is Momentum Extragradient Optimal? A Polynomial-Based Analysis
by: Kim, Junhyung Lyle, et al.
Published: (2022)

On the Error-Propagation of Inexact Hotelling's Deflation for Principal Component Analysis
by: Liao, Fangshuo, et al.
Published: (2023)

One Rank at a Time: Cascading Error Dynamics in Sequential Learning
by: Vandchali, Mahtab Alizadeh, et al.
Published: (2025)

Unveiling Hidden Pivotal Players with GoalNet: A GNN-Based Soccer Player Evaluation System
by: Jiang, Jacky Hao, et al.
Published: (2025)

Towards a Better Theoretical Understanding of Independent Subnetwork Training
by: Shulgin, Egor, et al.
Published: (2023)

Exploiting Low-Rank Structure in Max-K-Cut Problems
by: Stevens, Ria, et al.
Published: (2026)

GHOST: Unmasking Phantom States in Mamba2 via Grouped Hidden-state Output-aware Selection & Truncation
by: Menezes, Michael, et al.
Published: (2026)

Quantum EigenGame for excited state calculation
by: Quiroga, David, et al.
Published: (2025)

A Unified Kantorovich Duality for Multimarginal Optimal Transport
by: Cheryala, Yehya, et al.
Published: (2026)

A Catalyst Framework for the Quantum Linear System Problem via the Proximal Point Algorithm
by: Kim, Junhyung Lyle, et al.
Published: (2024)

RecCrysFormer: Refined Protein Structural Prediction from 3D Patterson Maps via Recycling Training Runs
by: Pan, Tom, et al.
Published: (2025)

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
by: He, Jianliang, et al.
Published: (2026)

Adaptive Federated Learning with Auto-Tuned Clients
by: Kim, Junhyung Lyle, et al.
Published: (2023)

Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
by: Kolomvaki, Afroditi, et al.
Published: (2026)

Thinking Out of the Box: Hybrid SAT Solving by Unconstrained Continuous Optimization
by: Zhang, Zhiwei, et al.
Published: (2025)

Using non-convex optimization in quantum process tomography: Factored gradient descent is tough to beat
by: Quiroga, David A., et al.
Published: (2023)

Online Residual Learning from Offline Experts for Pedestrian Tracking
by: Vlachos, Anastasios, et al.
Published: (2024)

An Optimal Control Approach To Transformer Training
by: Akman, Kağan, et al.
Published: (2026)

Achieving $ε^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions
by: Hamza, Ishaq, et al.
Published: (2026)

Finite Sample Frequency Domain Identification
by: Tsiamis, Anastasios, et al.
Published: (2024)

Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence
by: Zhang, Shiyuan, et al.
Published: (2026)

Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
by: Yang, Yufeng, et al.
Published: (2024)

Predictive Linear Online Tracking for Unknown Targets
by: Tsiamis, Anastasios, et al.
Published: (2024)

Training Infinitely Deep and Wide Transformers
by: Barboni, Raphaël, et al.
Published: (2026)

Schrödinger Bridge Over A Compact Connected Lie Group
by: Mahmood, Hamza, et al.
Published: (2026)

Online Optimization on Hadamard Manifolds: Curvature Independent Regret Bounds on Horospherically Convex Objectives
by: Sahinoglu, Emre, et al.
Published: (2025)

Gradient Equilibrium in Online Learning: Theory and Applications
by: Angelopoulos, Anastasios N., et al.
Published: (2025)

Mean-Field Limits for Two-Layer Neural Networks Trained with Consensus-Based Optimization
by: De Deyn, William, et al.
Published: (2025)

DOGE-Train: Discrete Optimization on GPU with End-to-end Training
by: Abbas, Ahmed, et al.
Published: (2022)

Assessing and Enhancing Graph Neural Networks for Combinatorial Optimization: Novel Approaches and Application in Maximum Independent Set Problems
by: Hu, Chenchuhui
Published: (2024)

Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systems
by: Hu, Junkai, et al.
Published: (2025)

A Compositional Kernel Model for Feature Learning
by: Ruan, Feng, et al.
Published: (2025)

Finite-sample guarantees for data-driven forward-backward operator methods
by: Fabiani, Filippo, et al.
Published: (2025)