Cover Image

Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	He, Ziwei, Yuan, Jian, Zhou, Le, Leng, Jingwen, Jiang, Bo
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
by: Yuan, Jian, et al.
Published: (2025)

TreeKV: Smooth Key-Value Cache Compression with Tree Structures
by: He, Ziwei, et al.
Published: (2025)

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
by: He, Ziwei, et al.
Published: (2023)

$π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling
by: Liu, Dong, et al.
Published: (2025)

Training-free Context-adaptive Attention for Efficient Long Context Modeling
by: You, Zeng, et al.
Published: (2025)

Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
by: Yan, Siyuan, et al.
Published: (2025)

Latent-Condensed Transformer for Efficient Long Context Modeling
by: You, Zeng, et al.
Published: (2026)

CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information
by: Wang, Yuxin, et al.
Published: (2024)

Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
by: Fei, Weizhi, et al.
Published: (2025)

Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models
by: Molfese, Francesco Maria, et al.
Published: (2026)

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
by: Chen, Yingfa, et al.
Published: (2026)

Cost-Optimal Grouped-Query Attention for Long-Context Modeling
by: Chen, Yingfa, et al.
Published: (2025)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
by: MiniCPM Team, et al.
Published: (2026)

Multipole Attention for Efficient Long Context Reasoning
by: Hooper, Coleman, et al.
Published: (2025)

Thus Spake Long-Context Large Language Model
by: Liu, Xiaoran, et al.
Published: (2025)

COMI: Coarse-to-fine Context Compression via Marginal Information Gain
by: Tang, Jiwei, et al.
Published: (2026)

Context as a Tool: Context Management for Long-Horizon SWE-Agents
by: Liu, Shukai, et al.
Published: (2025)

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
by: Zarch, Hossein Entezari, et al.
Published: (2025)

Reranking Passages with Coarse-to-Fine Neural Retriever Enhanced by List-Context Information
by: Zhu, Hongyin
Published: (2023)

Efficient Context Scaling with LongCat ZigZag Attention
by: Zhang, Chen, et al.
Published: (2025)

LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
by: Liu, Xiaoran, et al.
Published: (2025)

EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention
by: Shi, Yulong, et al.
Published: (2023)

HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing
by: He, Zifan, et al.
Published: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
by: Lu, Yi, et al.
Published: (2024)

LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
by: Liu, Zeyu, et al.
Published: (2025)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
by: Ling Team, et al.
Published: (2025)

Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
by: Hu, Xiang, et al.
Published: (2025)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
by: Chen, Yukang, et al.
Published: (2023)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
by: Li, Wenxuan, et al.
Published: (2025)

Long Context RAG Performance of Large Language Models
by: Leng, Quinn, et al.
Published: (2024)

ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
by: Liu, Zhuorui, et al.
Published: (2025)

HyLRA: Hybrid Layer Reuse Attention for Efficient Long-Context Inference
by: Ai, Xuan, et al.
Published: (2026)

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
by: Xiao, Guangxuan, et al.
Published: (2024)

MoBA: Mixture of Block Attention for Long-Context LLMs
by: Lu, Enzhe, et al.
Published: (2025)

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
by: Hu, Zhiyuan, et al.
Published: (2024)

Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
by: Zhao, Yusheng, et al.
Published: (2026)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
by: Hu, Xiang, et al.
Published: (2024)

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
by: Gu, Zhuohan, et al.
Published: (2024)

Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention
by: Zhan, Zhihao, et al.
Published: (2025)

FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
by: Lai, Xunhao, et al.
Published: (2025)