Image de couverture de livre

Scaling Context Requires Rethinking Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Gelada, Carles, Buckman, Jacob, Zhang, Sean, Bach, Txus
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Conformal Transformations for Symmetric Power Transformers
par: Kumar, Saurabh, et autres
Publié: (2025)

Which Attention Heads Matter for In-Context Learning?
par: Yin, Kayo, et autres
Publié: (2025)

Rethinking Early Stopping: Refine, Then Calibrate
par: Berta, Eugène, et autres
Publié: (2025)

Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
par: Bouadi, Mohamed, et autres
Publié: (2025)

Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scaling
par: Xia, Fanzeng, et autres
Publié: (2025)

Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluation
par: Kong, Yaxuan, et autres
Publié: (2025)

UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
par: Al-Tahan, Haider, et autres
Publié: (2024)

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
par: Lee, Changhun, et autres
Publié: (2025)

Stem: Rethinking Causal Information Flow in Sparse Attention
par: Niu, Lin, et autres
Publié: (2026)

Scalable Context-Aware Graph Attention for Unsupervised Anomaly Detection in Large-Scale Mobile Networks
par: Malacarne, Sara, et autres
Publié: (2026)

Continued AI Scaling Requires Repeated Efficiency Doublings
par: Lu, Chien-Ping
Publié: (2026)

Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inference
par: Wilkinghoff, Kevin, et autres
Publié: (2026)

When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning
par: Qiu, Chenghao, et autres
Publié: (2026)

Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
par: Feng, Yunzhen, et autres
Publié: (2024)

Long-Context Attention Benchmark: From Kernel Efficiency to Distributed Context Parallelism
par: Bu, Tao, et autres
Publié: (2025)

Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series
par: Yu, Guoqi, et autres
Publié: (2026)

Rethinking Zero-Shot Time Series Classification: From Task-specific Classifiers to In-Context Inference
par: Fang, Juntao, et autres
Publié: (2026)

Structured Matrix Scaling for Multi-Class Calibration
par: Berta, Eugène, et autres
Publié: (2025)

MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
par: Zhou, Ruijie, et autres
Publié: (2026)

Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
par: Joshi, Thomas, et autres
Publié: (2025)

Attention in Constant Time: Vashista Sparse Attention for Long-Context Decoding with Exponential Guarantees
par: Nobaub, Vashista
Publié: (2026)

Indirect Attention: Turning Context Misalignment into a Feature
par: Bahaduri, Bissmella, et autres
Publié: (2025)

Superiority of Multi-Head Attention in In-Context Linear Regression
par: Cui, Yingqian, et autres
Publié: (2024)

Scaling Attention via Feature Sparsity
par: Xie, Yan, et autres
Publié: (2026)

RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts
par: Joshi, Sahil, et autres
Publié: (2025)

Rethinking Optimal Verification Granularity for Compute-Efficient Test-Time Scaling
par: Chen, Hao Mark, et autres
Publié: (2025)

CalArena: A Large-Scale Post-Hoc Calibration Benchmark
par: Berta, Eugène, et autres
Publié: (2026)

Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
par: Qiao, Ye, et autres
Publié: (2025)

Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
par: Ni, Wentao, et autres
Publié: (2026)

Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning
par: Bouadi, Mohamed, et autres
Publié: (2025)

Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow
par: Delestre, Cyrile, et autres
Publié: (2024)

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
par: Liu, Dong, et autres
Publié: (2026)

Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models
par: Wen, Ziting, et autres
Publié: (2024)

The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
par: Karten, Seth, et autres
Publié: (2026)

Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering
par: Zhou, Han, et autres
Publié: (2023)

Efficient Low Rank Attention for Long-Context Inference in Large Language Models
par: Li, Tenghui, et autres
Publié: (2025)

VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
par: Guanzhong, Chen
Publié: (2026)

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
par: Xu, Hongtao, et autres
Publié: (2026)

Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
par: Wang, Fan, et autres
Publié: (2025)

TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
par: Dong, Perry, et autres
Publié: (2026)