Image de couverture de livre

Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Bick, Aviv, Li, Kevin Y., Xing, Eric P., Kolter, J. Zico, Gu, Albert
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Retrieval-Aware Distillation for Transformer-SSM Hybrids
par: Bick, Aviv, et autres
Publié: (2026)

Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
par: Bick, Aviv, et autres
Publié: (2025)

Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
par: Bick, Aviv, et autres
Publié: (2025)

Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners
par: Paliotta, Daniele, et autres
Publié: (2025)

Mamba-3: Improved Sequence Modeling using State Space Principles
par: Lahoti, Aakash, et autres
Publié: (2026)

Mimetic Initialization of MLPs
par: Trockman, Asher, et autres
Publié: (2026)

Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
par: Duan, Xintong, et autres
Publié: (2025)

Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
par: Li, Kevin Y., et autres
Publié: (2024)

Weight Ensembling Improves Reasoning in Language Models
par: Dang, Xingyu, et autres
Publié: (2025)

A Simple and Effective Pruning Approach for Large Language Models
par: Sun, Mingjie, et autres
Publié: (2023)

One-Step Diffusion Distillation through Score Implicit Matching
par: Luo, Weijian, et autres
Publié: (2024)

Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line
par: Kim, Eungyeup, et autres
Publié: (2023)

Superhuman AI for Stratego Using Self-Play Reinforcement Learning and Test-Time Search
par: Sokota, Samuel, et autres
Publié: (2025)

Looking beyond the next token
par: Thankaraj, Abitha, et autres
Publié: (2025)

Base Models Look Human To AI Detectors
par: Xu, Yixuan Even, et autres
Publié: (2026)

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
par: Xu, Yixuan Even, et autres
Publié: (2025)

The Update-Equivalence Framework for Decision-Time Planning
par: Sokota, Samuel, et autres
Publié: (2023)

Compute-Optimal LLMs Provably Generalize Better With Scale
par: Finzi, Marc, et autres
Publié: (2025)

Provably Bounding Neural Network Preimages
par: Kotha, Suhas, et autres
Publié: (2023)

ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use
par: Tien, Jeremy, et autres
Publié: (2026)

Neural Network Verification with Branch-and-Bound for General Nonlinearities
par: Shi, Zhouxing, et autres
Publié: (2024)

Contextures: Representations from Contexts
par: Zhai, Runtian, et autres
Publié: (2025)

On the Expressive Power and Limitations of Multi-Layer SSMs
par: Zubić, Nikola, et autres
Publié: (2026)

Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions
par: Pei, Yan Ru
Publié: (2025)

Understanding Optimization in Deep Learning with Central Flows
par: Cohen, Jeremy M., et autres
Publié: (2024)

Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification
par: Farsang, Mónika, et autres
Publié: (2026)

Existing Large Language Model Unlearning Evaluations Are Inconclusive
par: Feng, Zhili, et autres
Publié: (2025)

One-Step Diffusion Distillation via Deep Equilibrium Models
par: Geng, Zhengyang, et autres
Publié: (2023)

Delta Knowledge Distillation for Large Language Models
par: Cao, Yihan, et autres
Publié: (2025)

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
par: Dao, Tri, et autres
Publié: (2024)

Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion
par: Dontas, Michail, et autres
Publié: (2024)

Training a Generally Curious Agent
par: Tajwar, Fahim, et autres
Publié: (2025)

Antidistillation Fingerprinting
par: Xu, Yixuan Even, et autres
Publié: (2026)

Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach
par: Duan, Zhihua, et autres
Publié: (2025)

Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions
par: Li, Zuguang, et autres
Publié: (2024)

Practical Insights into Knowledge Distillation for Pre-Trained Models
par: Alballa, Norah, et autres
Publié: (2024)

Model Merging via Multi-Teacher Knowledge Distillation
par: Dalili, Seyed Arshan, et autres
Publié: (2025)

Mamba: Linear-Time Sequence Modeling with Selective State Spaces
par: Gu, Albert, et autres
Publié: (2023)

Enhancing Knowledge Graph Completion with GNN Distillation and Probabilistic Interaction Modeling
par: Wang, Lingzhi, et autres
Publié: (2025)

When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time?
par: Li, Chenyang, et autres
Publié: (2025)