Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913846597255168 |
|---|---|
| author | Cheng, Zifeng Wang, Zhonghui Fu, Yuchen Jiang, Zhiwei Yin, Yafeng Wang, Cong Gu, Qing |
| author_facet | Cheng, Zifeng Wang, Zhonghui Fu, Yuchen Jiang, Zhiwei Yin, Yafeng Wang, Cong Gu, Qing |
| contents | Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12831 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering Cheng, Zifeng Wang, Zhonghui Fu, Yuchen Jiang, Zhiwei Yin, Yafeng Wang, Cong Gu, Qing Computation and Language Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP. |
| title | Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.12831 |