Cover Image

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Shuaiyi, Zhang, Zhisong, Wang, Yan, Zhu, Lei, Ma, Dongyang, Deng, Chenlong, Deng, Yang, Lam, Wai
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text
by: Li, Shuaiyi, et al.
Published: (2023)

Block-Attention for Efficient Prefilling
by: Ma, Dongyang, et al.
Published: (2024)

On the Worst Prompt Performance of Large Language Models
by: Cao, Bowen, et al.
Published: (2024)

InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing
by: Li, Shuaiyi, et al.
Published: (2025)

InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation
by: Cao, Bowen, et al.
Published: (2025)

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
by: Jiang, Yuxuan, et al.
Published: (2026)

A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression
by: Deng, Chenlong, et al.
Published: (2024)

Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models
by: Zhang, Zhisong, et al.
Published: (2024)

GLARE: Agentic Reasoning for Legal Judgment Prediction
by: Yang, Xinyu, et al.
Published: (2025)

UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
by: Deng, Chenlong, et al.
Published: (2025)

DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
by: Gurses, Selcuk, et al.
Published: (2025)

Sparser Block-Sparse Attention via Token Permutation
by: Wang, Xinghao, et al.
Published: (2025)

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt
by: Cai, Deng, et al.
Published: (2024)

X-Blocks: Linguistic Building Blocks of Natural Language Explanations for Automated Vehicles
by: Zadeh, Ashkan Y., et al.
Published: (2026)

MoBA: Mixture of Block Attention for Long-Context LLMs
by: Lu, Enzhe, et al.
Published: (2025)

FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
by: Chen, Zhuokun, et al.
Published: (2026)

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
by: Deng, Yang, et al.
Published: (2023)

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
by: Yang, Chuanpeng, et al.
Published: (2024)

The End of Manual Decoding: Towards Truly End-to-End Language Models
by: Wang, Zhichao, et al.
Published: (2025)

Prism: Spectral-Aware Block-Sparse Attention
by: Wang, Xinghao, et al.
Published: (2026)

BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization
by: Li, Ji-Fu, et al.
Published: (2026)

Consecutive Batch Model Editing with HooK Layers
by: Li, Shuaiyi, et al.
Published: (2024)

Towards Better Text-to-Image Generation Alignment via Attention Modulation
by: Wu, Yihang, et al.
Published: (2024)

Neural Attention Search
by: Deng, Difan, et al.
Published: (2025)

Distilling to Hybrid Attention Models via KL-Guided Layer Selection
by: Li, Yanhong, et al.
Published: (2025)

CORBA: Contagious Recursive Blocking Attacks on Multi-Agent Systems Based on Large Language Models
by: Zhou, Zhenhong, et al.
Published: (2025)

A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers
by: Zhao, Xiaoyan, et al.
Published: (2023)

Multi-LLM Collaborative Search for Complex Problem Solving
by: Yang, Sen, et al.
Published: (2025)

Atomic Calibration of LLMs in Long-Form Generations
by: Zhang, Caiqi, et al.
Published: (2024)

EBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMs
by: Guo, Song, et al.
Published: (2024)

Self-Correction Distillation for Structured Data Question Answering
by: Zhu, Yushan, et al.
Published: (2025)

Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments
by: Latif, Ehsan, et al.
Published: (2023)

Transformer Block Coupling and its Correlation with Generalization in LLMs
by: Aubry, Murdock, et al.
Published: (2024)

Entropy-Based Block Pruning for Efficient Large Language Models
by: Yang, Liangwei, et al.
Published: (2025)

LoGU: Long-form Generation with Uncertainty Expressions
by: Yang, Ruihan, et al.
Published: (2024)

Exploring the System 1 Thinking Capability of Large Reasoning Models
by: Zhang, Wenyuan, et al.
Published: (2025)

Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations
by: Qin, Peixin, et al.
Published: (2024)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
by: Deng, Yichuan, et al.
Published: (2024)

GTA: Grouped-head latenT Attention
by: Sun, Luoyang, et al.
Published: (2025)

Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
by: Zhu, Xunyu, et al.
Published: (2024)