Cover Image

Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xia, Xiaojie, Zhang, Huigang, Zhong, Chaoliang, Sun, Jun, Oishi, Yusuke
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Following the Teacher's Footsteps: Scheduled Checkpoint Distillation for Domain-Specific LLMs
by: Feng, Cheng, et al.
Published: (2026)

From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
by: Ren, Yuxin, et al.
Published: (2026)

Native Hybrid Attention for Efficient Sequence Modeling
by: Du, Jusen, et al.
Published: (2025)

Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement
by: Yang, Shu, et al.
Published: (2025)

BrainDistill: Implantable Motor Decoding with Task-Specific Knowledge Distillation
by: Xie, Yuhan, et al.
Published: (2026)

GSTAM: Efficient Graph Distillation with Structural Attention-Matching
by: Rasti-Meymandi, Arash, et al.
Published: (2024)

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
by: Chen, Yingfa, et al.
Published: (2026)

Efficient Multi-Task Modeling through Automated Fusion of Trained Models
by: Zhou, Jingxuan, et al.
Published: (2025)

Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation
by: Dey, Vishal, et al.
Published: (2024)

Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
by: De Muri, Giovanni, et al.
Published: (2025)

The Geometry of Self-Verification in a Task-Specific Reasoning Model
by: Lee, Andrew, et al.
Published: (2025)

ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks
by: Saini, Dhruv, et al.
Published: (2026)

EffiCANet: Efficient Time Series Forecasting with Convolutional Attention
by: Zhou, Xinxing, et al.
Published: (2024)

DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher
by: Zhong, Yisheng, et al.
Published: (2026)

The Mamba in the Llama: Distilling and Accelerating Hybrid Models
by: Wang, Junxiong, et al.
Published: (2024)

CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
by: Yang, Yibo, et al.
Published: (2024)

CADENT: Gated Hybrid Distillation for Sample-Efficient Transfer in Reinforcement Learning
by: Alinejad, Mahyar, et al.
Published: (2026)

Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correction
by: Feng, Jinyuan, et al.
Published: (2024)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
by: MiniCPM Team, et al.
Published: (2026)

Easy Adaptation: An Efficient Task-Specific Knowledge Injection Method for Large Models in Resource-Constrained Environments
by: Chen, Dong, et al.
Published: (2025)

Aligning Knowledge Graphs Provided by Humans and Generated from Neural Networks in Specific Tasks
by: Li, Tangrui, et al.
Published: (2024)

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications
by: Wang, Jun, et al.
Published: (2023)

Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
by: Shen, Yuhao, et al.
Published: (2026)

FISformer: Replacing Self-Attention with a Fuzzy Inference System in Transformer Models for Time Series Forecasting
by: Haznedar, Bulent, et al.
Published: (2026)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
by: Ling Team, et al.
Published: (2025)

Online Policy Distillation with Decision-Attention
by: Yu, Xinqiang, et al.
Published: (2024)

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
by: Fan, Tao, et al.
Published: (2025)

DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL
by: Jackermeier, Mathias, et al.
Published: (2024)

Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling
by: Li, Mingzhuo, et al.
Published: (2025)

LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid
by: Sun, Weigao, et al.
Published: (2025)

InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
by: Wang, Youjin, et al.
Published: (2026)

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context
by: Augustin, Maximilian, et al.
Published: (2024)

Data-Efficient Symbolic Regression via Foundation Model Distillation
by: Ying, Wangyang, et al.
Published: (2025)

HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
by: Ding, Ken
Published: (2026)

Speculative Coreset Selection for Task-Specific Fine-tuning
by: Zhang, Xiaoyu, et al.
Published: (2024)

Resource-Efficient Iterative LLM-Based NAS with Feedback Memory
by: Gu, Xiaojie, et al.
Published: (2026)

Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
by: Li, Derek, et al.
Published: (2025)

Distilling Time Series Foundation Models for Efficient Forecasting
by: Li, Yuqi, et al.
Published: (2026)

Retrieval-Aware Distillation for Transformer-SSM Hybrids
by: Bick, Aviv, et al.
Published: (2026)

Mixture of Layers with Hybrid Attention
by: Ternovtsii, Ivan, et al.
Published: (2026)