HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910233828261888 |
|---|---|
| author | Wang, Juyuan Wang, Chenxing Fang, Yuchen Hu, Huiyun Du, Junwu Li, Aolin Rong, Shunlin Wu, Haijun Xu, Jin Liu, Ligang Liao, Dongliang |
| author_facet | Wang, Juyuan Wang, Chenxing Fang, Yuchen Hu, Huiyun Du, Junwu Li, Aolin Rong, Shunlin Wu, Haijun Xu, Jin Liu, Ligang Liao, Dongliang |
| contents | Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to $\mathcal{O}(1)$ forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank achieves the highest average NDCG@10 at every scale, outperforming both generative and decoding-free baselines on the majority of benchmarks with 100\% formatting success. At 4B, 57.4\% of relevant middle-zone documents reach the top quartile versus 14.2\% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17237 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads Wang, Juyuan Wang, Chenxing Fang, Yuchen Hu, Huiyun Du, Junwu Li, Aolin Rong, Shunlin Wu, Haijun Xu, Jin Liu, Ligang Liao, Dongliang Information Retrieval Artificial Intelligence Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to $\mathcal{O}(1)$ forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank achieves the highest average NDCG@10 at every scale, outperforming both generative and decoding-free baselines on the majority of benchmarks with 100\% formatting success. At 4B, 57.4\% of relevant middle-zone documents reach the top quartile versus 14.2\% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking. |
| title | HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2604.17237 |