Prism-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912961602256896 |
|---|---|
| author | Ge, Yuyao Liu, Shenghua Wang, Yiwei Liu, Tianyu Bi, Baolong Mei, Lingrui Yao, Jiayu Guo, Jiafeng Cheng, Xueqi |
| author_facet | Ge, Yuyao Liu, Shenghua Wang, Yiwei Liu, Tianyu Bi, Baolong Mei, Lingrui Yao, Jiayu Guo, Jiafeng Cheng, Xueqi |
| contents | Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, PRISM-$Δ$ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the fluency cost of steering. PRISM-$Δ$ also scales to long-context retrieval, outperforming the best existing method by up to +4.8% relative gain. PRISM-$Δ$ is compatible with FlashAttention and adds negligible memory overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10705 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Prism-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models Ge, Yuyao Liu, Shenghua Wang, Yiwei Liu, Tianyu Bi, Baolong Mei, Lingrui Yao, Jiayu Guo, Jiafeng Cheng, Xueqi Computation and Language Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, PRISM-$Δ$ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the fluency cost of steering. PRISM-$Δ$ also scales to long-context retrieval, outperforming the best existing method by up to +4.8% relative gain. PRISM-$Δ$ is compatible with FlashAttention and adds negligible memory overhead. |
| title | Prism-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.10705 |