Imagen de Portada

State Space Models are Provably Comparable to Transformers in Dynamic Token Selection

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Nishikawa, Naoki, Suzuki, Taiji
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
por: Kawata, Ryotaro, et al.
Publicado: (2025)

Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
por: Nishikawa, Naoki, et al.
Publicado: (2025)

Transformers Provably Solve Parity Efficiently with Chain of Thought
por: Kim, Juno, et al.
Publicado: (2024)

From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
por: Kawata, Ryotaro, et al.
Publicado: (2025)

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
por: Wakayama, Tomoya, et al.
Publicado: (2025)

Direct Distributional Optimization for Provable Alignment of Diffusion Models
por: Kawata, Ryotaro, et al.
Publicado: (2025)

Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
por: Kim, Juno, et al.
Publicado: (2024)

Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation
por: Kim, Juno, et al.
Publicado: (2025)

Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
por: Bu, Dake, et al.
Publicado: (2025)

Transformers as Measure-Theoretic Associative Memory: A Statistical Perspective and Minimax Optimality
por: Kawata, Ryotaro, et al.
Publicado: (2026)

Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input
por: Takakura, Shokichi, et al.
Publicado: (2023)

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning
por: Bu, Dake, et al.
Publicado: (2024)

Demystifying the Token Dynamics of Deep Selective State Space Models
por: Vo, Thieu N, et al.
Publicado: (2024)

Transformers are Minimax Optimal Nonparametric In-Context Learners
por: Kim, Juno, et al.
Publicado: (2024)

From Saddle Points Toward Global Minima: A Newton-Type Method on Wasserstein Space
por: Lascu, Razvan-Andrei, et al.
Publicado: (2026)

Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine
por: Huang, Wei, et al.
Publicado: (2026)

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
por: Higuchi, Rei, et al.
Publicado: (2025)

Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective
por: Takakura, Shokichi, et al.
Publicado: (2024)

Deep Two-Way Matrix Reordering for Relational Data Analysis
por: Watanabe, Chihiro, et al.
Publicado: (2021)

The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge
por: Awano, Ryoya, et al.
Publicado: (2026)

AutoLL: Automatic Linear Layout of Graphs based on Deep Neural Network
por: Watanabe, Chihiro, et al.
Publicado: (2021)

Test time training enhances in-context learning of nonlinear functions
por: Kuwataka, Kento, et al.
Publicado: (2025)

Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples
por: Bu, Dake, et al.
Publicado: (2024)

Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot
por: Wang, Zixuan, et al.
Publicado: (2024)

Provable In-Context Vector Arithmetic via Retrieving Task Concepts
por: Bu, Dake, et al.
Publicado: (2025)

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
por: Higuchi, Rei, et al.
Publicado: (2025)

Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
por: Jiang, Jiarui, et al.
Publicado: (2024)

MambaByte: Token-free Selective State Space Model
por: Wang, Junxiong, et al.
Publicado: (2024)

Attention with Trained Embeddings Provably Selects Important Tokens
por: Wu, Diyuan, et al.
Publicado: (2025)

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
por: Joseph, Federico Arangath, et al.
Publicado: (2024)

Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
por: Oh, Junsoo, et al.
Publicado: (2025)

How do Transformers perform In-Context Autoregressive Learning?
por: Sander, Michael E., et al.
Publicado: (2024)

Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
por: Nitanda, Atsushi, et al.
Publicado: (2025)

Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
por: Götz, Leon, et al.
Publicado: (2024)

Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
por: Uesaka, Toshimitsu, et al.
Publicado: (2024)

State-Free Inference of State-Space Models: The Transfer Function Approach
por: Parnichkun, Rom N., et al.
Publicado: (2024)

DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models
por: Bu, Dake, et al.
Publicado: (2026)

Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
por: Kinoshita, Yuri, et al.
Publicado: (2026)

Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds
por: Fu, Guoji, et al.
Publicado: (2026)

Convergence Error Analysis of Reflected Gradient Langevin Dynamics for Globally Optimizing Non-Convex Constrained Problems
por: Sato, Kanji, et al.
Publicado: (2022)