PromptEmbedder:: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913166881980416 |
|---|---|
| author | Tsai, Yu-Che Chen, Kuan-Yu Chen, Yuan-Hao Chang, Yu-Han Tsai, Ching-Yu Chuang, Yu-Hsiang Lin, Shou-De |
| author_facet | Tsai, Yu-Che Chen, Kuan-Yu Chen, Yuan-Hao Chang, Yu-Han Tsai, Ching-Yu Chuang, Yu-Hsiang Lin, Shou-De |
| contents | Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability. Whenever a new backbone emerges, existing approaches require costly retraining from scratch. To address this, we propose PromptEmbedder, a novel dual-LLM framework that decouples embedding knowledge from specific backbone weights. PromptEmbedder utilizes a Prompting LLM to generate instruction-aware soft prompts for a frozen Embedding LLM via a differentiable generation process with continuous relaxation, ensuring full gradient flow during contrastive training. By localizing task-specific knowledge within the Prompting LLM, adapting to new architectures requires only retraining a lightweight linear alignment matrix. Evaluations on the MTEB benchmark show that PromptEmbedder achieves comparable performance with LoRA finetuning while reducing GPU memory by 40% and accelerating training by 3.7x. Our approach establishes a scalable, architecture-agnostic paradigm for efficient LLM-based representation learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28066 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PromptEmbedder:: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting Tsai, Yu-Che Chen, Kuan-Yu Chen, Yuan-Hao Chang, Yu-Han Tsai, Ching-Yu Chuang, Yu-Hsiang Lin, Shou-De Computation and Language Artificial Intelligence Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability. Whenever a new backbone emerges, existing approaches require costly retraining from scratch. To address this, we propose PromptEmbedder, a novel dual-LLM framework that decouples embedding knowledge from specific backbone weights. PromptEmbedder utilizes a Prompting LLM to generate instruction-aware soft prompts for a frozen Embedding LLM via a differentiable generation process with continuous relaxation, ensuring full gradient flow during contrastive training. By localizing task-specific knowledge within the Prompting LLM, adapting to new architectures requires only retraining a lightweight linear alignment matrix. Evaluations on the MTEB benchmark show that PromptEmbedder achieves comparable performance with LoRA finetuning while reducing GPU memory by 40% and accelerating training by 3.7x. Our approach establishes a scalable, architecture-agnostic paradigm for efficient LLM-based representation learning. |
| title | PromptEmbedder:: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.28066 |