Cover Image

Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ratajczak, Martin, Robichaud, Jean-Philippe, Fox, Jennifer Drexler
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Reverb: Open-Source ASR and Diarization from Rev
by: Bhandari, Nishchal, et al.
Published: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
by: Lu, Yi, et al.
Published: (2024)

MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads
by: Liu, Weihao, et al.
Published: (2025)

Knocking-Heads Attention
by: Zhou, Zhanchao, et al.
Published: (2025)

Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
by: Yang, Lijie, et al.
Published: (2025)

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
by: Donhauser, Konstantin, et al.
Published: (2025)

Segmental Attention Decoding With Long Form Acoustic Encodings
by: Swietojanski, Pawel, et al.
Published: (2025)

ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
by: Liu, Zhuorui, et al.
Published: (2025)

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
by: Xiao, Guangxuan, et al.
Published: (2024)

S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
by: Lin, Xihui, et al.
Published: (2024)

Latent Multi-Head Attention for Small Language Models
by: Mehta, Sushant, et al.
Published: (2025)

Analyzing Multi-Head Attention on Trojan BERT Models
by: Wang, Jingwei
Published: (2024)

Improving Transformers with Dynamically Composable Multi-Head Attention
by: Xiao, Da, et al.
Published: (2024)

ART: Attention Replacement Technique to Improve Factuality in LLMs
by: Luo, Ziqin, et al.
Published: (2026)

Improving ASR Contextual Biasing with Guided Attention
by: Tang, Jiyang, et al.
Published: (2024)

AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR
by: Beck, Eugen, et al.
Published: (2026)

Style-agnostic evaluation of ASR using multiple reference transcripts
by: McNamara, Quinten, et al.
Published: (2024)

Multi-Head Attention Is a Multi-Player Game
by: Chakrabarti, Kushal, et al.
Published: (2026)

Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
by: Kopiczko, Dawid J., et al.
Published: (2024)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
by: Musat, Tiberiu
Published: (2024)

CHAI: Clustered Head Attention for Efficient LLM Inference
by: Agarwal, Saurabh, et al.
Published: (2024)

AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
by: Hua, Kai, et al.
Published: (2025)

Interpreting Transformers Through Attention Head Intervention
by: Kadem, Mason, et al.
Published: (2026)

Inferring Functionality of Attention Heads from their Parameters
by: Elhelo, Amit, et al.
Published: (2024)

MossNet: Mixture of State-Space Experts is a Multi-Head Attention
by: Tuli, Shikhar, et al.
Published: (2025)

Detection of Opioid Users from Reddit Posts via an Attention-based Bidirectional Recurrent Neural Network
by: Wang, Yuchen, et al.
Published: (2024)

Sycophancy Hides Linearly in the Attention Heads
by: Genadi, Rifo, et al.
Published: (2026)

Recurrent Alignment with Hard Attention for Hierarchical Text Rating
by: Lin, Chenxi, et al.
Published: (2024)

Integrating Multi-Head Convolutional Encoders with Cross-Attention for Improved SPARQL Query Translation
by: Chen, Yi-Hui, et al.
Published: (2024)

Head-wise Shareable Attention for Large Language Models
by: Cao, Zouying, et al.
Published: (2024)

Attention Heads of Large Language Models: A Survey
by: Zheng, Zifan, et al.
Published: (2024)

Attention Mechanism and Heuristic Approach: Context-Aware File Ranking Using Multi-Head Self-Attention
by: Sharma, Pradeep Kumar, et al.
Published: (2026)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
by: Knupp, Jonas, et al.
Published: (2026)

Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
by: Sun, Bowen, et al.
Published: (2025)

Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks
by: Liu, Kai, et al.
Published: (2025)

RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
by: Liu, Siran, et al.
Published: (2026)

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
by: Gu, Yuxuan, et al.
Published: (2025)

Scaling Bidirectional Spans and Span Violations in Attention Mechanism
by: Kim, Jongwook, et al.
Published: (2025)

The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
by: Chiang, Ting-Rui, et al.
Published: (2025)

Surgical Repair of Collapsed Attention Heads in ALiBi Transformers
by: Schallon, Palmer
Published: (2026)