Cover Image

Short window attention enables long-term memorization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Cabannes, Loïc, Beck, Maximilian, Szilvasy, Gergely, Douze, Matthijs, Lomeli, Maria, Copet, Jade, Mazaré, Pierre-Emmanuel, Synnaeve, Gabriel, Jégou, Hervé
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Stochastic activations
by: Lomeli, Maria, et al.
Published: (2025)

Inference-time sparse attention with asymmetric indexing
by: Mazaré, Pierre-Emmanuel, et al.
Published: (2025)

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
by: Szilvasy, Gergely, et al.
Published: (2026)

Vector search with small radiuses
by: Szilvasy, Gergely, et al.
Published: (2024)

The Faiss library
by: Douze, Matthijs, et al.
Published: (2024)

Machine learning and high dimensional vector search
by: Douze, Matthijs
Published: (2025)

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
by: Gehring, Jonas, et al.
Published: (2024)

Simple and Controllable Music Generation
by: Copet, Jade, et al.
Published: (2023)

Functional Invariants to Watermark Large Transformers
by: Fernandez, Pierre, et al.
Published: (2023)

Towards a Neural Debugger for Python
by: Beck, Maximilian, et al.
Published: (2026)

SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
by: Wei, Yuxiang, et al.
Published: (2025)

Masked Audio Generation using a Single Non-Autoregressive Transformer
by: Ziv, Alon, et al.
Published: (2024)

Watermark Anything with Localized Messages
by: Sander, Tom, et al.
Published: (2024)

Watermarking Makes Language Models Radioactive
by: Sander, Tom, et al.
Published: (2024)

Qinco2: Vector Compression and Search with Improved Implicit Neural Codebooks
by: Vallaeys, Théophane, et al.
Published: (2025)

Moshi: a speech-text foundation model for real-time dialogue
by: Défossez, Alexandre, et al.
Published: (2024)

Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
by: Singh, Aaditya K., et al.
Published: (2024)

Textually Pretrained Speech Language Models
by: Hassid, Michael, et al.
Published: (2023)

Automatic Textbook Formalization
by: Gloeckle, Fabian, et al.
Published: (2026)

KVzap: Fast, Adaptive, and Faithful KV Cache Pruning
by: Jegou, Simon, et al.
Published: (2026)

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search
by: Severo, Daniel, et al.
Published: (2025)

Audio Conditioning for Music Generation via Discrete Bottleneck Features
by: Rouard, Simon, et al.
Published: (2024)

Successful reintroduction of species: improving on windows of opportunity for biodiversity repair
by: Ann‐Kathrin Tielke, et al.
Published: (2024)

An Independence-promoting Loss for Music Generation with Language Models
by: Lemercier, Jean-Marie, et al.
Published: (2024)

BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?
by: Chambon, Pierre, et al.
Published: (2025)

Neutral Residues: Revisiting Adapters for Model Extension
by: Talla, Franck Signe, et al.
Published: (2024)

Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
by: Devoto, Alessio, et al.
Published: (2025)

La hospitalidad en profesores memorables universitarios
by: Luis Gabriel Porta
Published: (2017)

Robust Offset-free Kernelized Data-Driven Predictive Control for Nonlinear Systems
by: Mazare, Mahmood, et al.
Published: (2025)

Noise-Tolerant Hybrid Approach for Data-Driven Predictive Control
by: Mazare, Mahmood, et al.
Published: (2025)

Short and long‐term effects of experimental varicocele
by: Aram Minas, et al.
Published: (2025)

TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
by: Jain, Kush, et al.
Published: (2024)

Getting the most out of your tokenizer for pre-training and domain adaptation
by: Dagan, Gautier, et al.
Published: (2024)

Surgical stabilization technique and long‐term outcome of traumatic lateral shoulder luxation in a dog
by: Maria Podsiedlik, et al.
Published: (2025)

Disentangling generalization and memorization in large language models using chess
by: Pleiss, Leonard S., et al.
Published: (2026)

Touring sampling with pushforward maps
by: Cabannes, Vivien, et al.
Published: (2023)

The Galerkin method beats Graph-Based Approaches for Spectral Algorithms
by: Cabannes, Vivien, et al.
Published: (2023)

RA-DIT: Retrieval-Augmented Dual Instruction Tuning
by: Lin, Xi Victoria, et al.
Published: (2023)

Residual Quantization with Implicit Neural Codebooks
by: Huijben, Iris A. M., et al.
Published: (2024)

Losing dimensions: Geometric memorization in generative diffusion
by: Achilli, Beatrice, et al.
Published: (2024)