Cover Image

Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Forchheimer, Robert
Format:	Preprint
Published:	2026
Subjects:	Machine Learning I.2.7
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors
by: Atad, Ido Andrew, et al.
Published: (2026)

The Hidden Attention of Mamba Models
by: Ali, Ameen, et al.
Published: (2024)

Merge-Bench: Resolve Merge Conflicts with Large Language Models
by: Schesch, Benedikt, et al.
Published: (2026)

RAPTOR-AI for Disaster OODA Loop: Hierarchical Multimodal RAG with Experience-Driven Agentic Decision-Making
by: Yasuno, Takato
Published: (2026)

Recurrent Memory-Augmented Transformers with Chunked Attention for Long-Context Language Modeling
by: Kashyap, Ankit
Published: (2025)

Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability
by: Bakish, Yarden, et al.
Published: (2025)

Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales
by: Salfati, Samuel
Published: (2026)

Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
by: Moghadasi, Mahdi Naser, et al.
Published: (2026)

QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
by: Zhou, Qirui, et al.
Published: (2025)

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
by: Mulgund, Abhijeet, et al.
Published: (2025)

OCRR: A Benchmark for Online Correction Recovery under Distribution Shift
by: Grassi, Adrian
Published: (2026)

Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text
by: Oketunji, Abiodun Finbarrs
Published: (2023)

TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax
by: Nauen, Tobias Christian, et al.
Published: (2024)

Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing
by: Sun, Yuhui, et al.
Published: (2025)

Combining Language and Topic Models for Hierarchical Text Classification
by: Toit, Jaco du, et al.
Published: (2025)

Chain and Causal Attention for Efficient Entity Tracking
by: Fagnou, Erwan, et al.
Published: (2024)

Introducing Three New Benchmark Datasets for Hierarchical Text Classification
by: Toit, Jaco du, et al.
Published: (2024)

Random Heterogeneous Neurochaos Learning Architecture for Data Classification
by: S, Remya Ajai A, et al.
Published: (2024)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
by: Fagnou, Erwan, et al.
Published: (2026)

Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
by: Zhao, Hangyue, et al.
Published: (2026)

Attention Drift: What Autoregressive Speculative Decoding Models Learn
by: Eldenk, Doğaç, et al.
Published: (2026)

Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
by: Gao, Heyang, et al.
Published: (2025)

Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
by: Zimerman, Itamar, et al.
Published: (2024)

On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
by: Zhao, Rosie, et al.
Published: (2026)

LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
by: Wang, Changsheng, et al.
Published: (2025)

Procedural Environment Generation for Tool-Use Agents
by: Sullivan, Michael, et al.
Published: (2025)

EvilGenie: A Reward Hacking Benchmark
by: Gabor, Jonathan, et al.
Published: (2025)

On the Challenges of Creating Datasets for Analyzing Commercial Sex Advertisements to Assess Human Trafficking Risk and Organized Activity
by: Rivas, Pablo, et al.
Published: (2024)

Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions
by: Soroka, Emi, et al.
Published: (2025)

Random Scaling of Emergent Capabilities
by: Zhao, Rosie, et al.
Published: (2025)

Spectral Clustering in Convex and Constrained Settings
by: Behera, Swarup Ranjan, et al.
Published: (2024)

Mr. Snuffleupagus at SemEval-2025 Task 4: Unlearning Factual Knowledge from LLMs Using Adaptive RMU
by: Dosajh, Arjun, et al.
Published: (2025)

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
by: Ahmadian, Arash, et al.
Published: (2024)

Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
by: Geuter, Jonathan, et al.
Published: (2025)

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
by: Zhao, Rosie, et al.
Published: (2025)

Interpreto: An Explainability Library for Transformers
by: Poché, Antonin, et al.
Published: (2025)

Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time
by: Zhao, Mingkuan, et al.
Published: (2026)

Beyond Memorization: Violating Privacy Via Inference with Large Language Models
by: Staab, Robin, et al.
Published: (2023)

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference
by: Dalal, Siddhartha, et al.
Published: (2024)

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
by: Szilvasy, Gergely, et al.
Published: (2026)