Imagen de Portada

Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Lin, Chaofan, Tang, Jiaming, Yang, Shuo, Wang, Hanshuo, Tang, Tian, Tian, Boyu, Stoica, Ion, Han, Song, Gao, Mingyu
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Post-Training Sparse Attention with Double Sparsity
por: Yang, Shuo, et al.
Publicado: (2024)

HashAttention: Semantic Sparsity for Faster Inference
por: Desai, Aditya, et al.
Publicado: (2024)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
por: Tang, Jiaming, et al.
Publicado: (2024)

BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
por: Xu, Peng, et al.
Publicado: (2024)

Frustratingly Easy Task-aware Pruning for Large Language Models
por: Tian, Yuanhe, et al.
Publicado: (2025)

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
por: Fu, Yichao, et al.
Publicado: (2024)

Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
por: Zhao, Yilong, et al.
Publicado: (2025)

DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
por: Huang, Yuxiang, et al.
Publicado: (2026)

A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention
por: Xiu, Di, et al.
Publicado: (2025)

LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
por: Yang, Shang, et al.
Publicado: (2025)

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
por: Wang, Hanrui, et al.
Publicado: (2020)

RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
por: Tang, Hanlin, et al.
Publicado: (2024)

DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
por: Gao, Shangqian, et al.
Publicado: (2024)

Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
por: Zhu, Kan, et al.
Publicado: (2025)

PocketLLM: Ultimate Compression of Large Language Models via Meta Networks
por: Tian, Ye, et al.
Publicado: (2025)

STS: Efficient Sparse Attention with Speculative Token Sparsity
por: Xu, Ceyu, et al.
Publicado: (2026)

Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
por: Tang, Zecheng, et al.
Publicado: (2026)

High-Layer Attention Pruning with Rescaling
por: Liu, Songtao, et al.
Publicado: (2025)

Adaptive Computation Pruning for the Forgetting Transformer
por: Lin, Zhixuan, et al.
Publicado: (2025)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
por: Tan, Sijun, et al.
Publicado: (2024)

GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning
por: Tian, Kaiyuan, et al.
Publicado: (2026)

Prompt-to-Leaderboard
por: Frick, Evan, et al.
Publicado: (2025)

Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment
por: Liu, Jun, et al.
Publicado: (2024)

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
por: Park, Jongseok, et al.
Publicado: (2026)

Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
por: Xi, Haocheng, et al.
Publicado: (2025)

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
por: Meng, Tian, et al.
Publicado: (2024)

SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
por: Liao, Huanxuan, et al.
Publicado: (2025)

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
por: Tian, Qiwei, et al.
Publicado: (2025)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
por: Cao, Mingyu, et al.
Publicado: (2024)

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
por: Xiao, Guangxuan, et al.
Publicado: (2024)

UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective
por: Xiong, Jing, et al.
Publicado: (2024)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
por: Liang, Yingyu, et al.
Publicado: (2024)

TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision
por: Zhang, Yunyi, et al.
Publicado: (2024)

Earley-Driven Dynamic Pruning for Efficient Structured Decoding
por: Sun, Xintong, et al.
Publicado: (2025)

R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
por: Jain, Naman, et al.
Publicado: (2025)

depyf: Open the Opaque Box of PyTorch Compiler for Machine Learning Researchers
por: You, Kaichao, et al.
Publicado: (2024)

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
por: Zhang, Jintao, et al.
Publicado: (2025)

DarwinLM: Evolutionary Structured Pruning of Large Language Models
por: Tang, Shengkun, et al.
Publicado: (2025)

SAP: Syntactic Attention Pruning for Transformer-based Language Models
por: Lee, Tzu-Yun, et al.
Publicado: (2025)

Efficient Vision-Language Reasoning via Adaptive Token Pruning
por: Li, Xue, et al.
Publicado: (2025)