LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915932160393216 |
|---|---|
| author | Xu, Dongjie Wu, Hao Shi, Weijie Cui, Yue Liu, Yuanjun Li, Jiawei Ma, Haolun Liu, An Zhu, Jia Xu, Jiajie |
| author_facet | Xu, Dongjie Wu, Hao Shi, Weijie Cui, Yue Liu, Yuanjun Li, Jiawei Ma, Haolun Liu, An Zhu, Jia Xu, Jiajie |
| contents | Through systematic experiments on long-context generation, we observe a damaging failure mode in which decoding can collapse into persistent repetition loops. We find that this degeneration is driven by collapsed attention patterns, where a subset of heads locks onto a narrow suffix of the history, and is further stabilized by inference-time KV cache reuse. Crucially, since many existing KV cache policies rely on attention-based importance, this collapse can produce spuriously high scores for repetitive tokens, causing cache management to inadvertently amplify repetition. To study this phenomenon in a controlled and reproducible manner, we introduce LoopBench, a benchmark with explicit loop-inducing conditions and loop-oriented metrics that quantify repetition severity and generation instability beyond downstream task scores. Building on these insights, we propose LoopGuard, a lightweight, plug-in KV cache guard that detects loop onset online and disrupts the feedback cycle by pruning repetitive tail spans under a fixed cache budget. Experiments on LoopBench show that LoopGuard reduces loop incidence by over 90 percentage points, while restoring output diversity and reducing token waste. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10044 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention Xu, Dongjie Wu, Hao Shi, Weijie Cui, Yue Liu, Yuanjun Li, Jiawei Ma, Haolun Liu, An Zhu, Jia Xu, Jiajie Artificial Intelligence Through systematic experiments on long-context generation, we observe a damaging failure mode in which decoding can collapse into persistent repetition loops. We find that this degeneration is driven by collapsed attention patterns, where a subset of heads locks onto a narrow suffix of the history, and is further stabilized by inference-time KV cache reuse. Crucially, since many existing KV cache policies rely on attention-based importance, this collapse can produce spuriously high scores for repetitive tokens, causing cache management to inadvertently amplify repetition. To study this phenomenon in a controlled and reproducible manner, we introduce LoopBench, a benchmark with explicit loop-inducing conditions and loop-oriented metrics that quantify repetition severity and generation instability beyond downstream task scores. Building on these insights, we propose LoopGuard, a lightweight, plug-in KV cache guard that detects loop onset online and disrupts the feedback cycle by pruning repetitive tail spans under a fixed cache budget. Experiments on LoopBench show that LoopGuard reduces loop incidence by over 90 percentage points, while restoring output diversity and reducing token waste. |
| title | LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.10044 |