Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.11124 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915734610771968 |
|---|---|
| author | Liang, Xiaoyu Peng, Yuchen Luo, Jiale Wang, Wenhao Hu, Haoji Zhou, Xincheng |
| author_facet | Liang, Xiaoyu Peng, Yuchen Luo, Jiale Wang, Wenhao Hu, Haoji Zhou, Xincheng |
| contents | Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11124 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings Liang, Xiaoyu Peng, Yuchen Luo, Jiale Wang, Wenhao Hu, Haoji Zhou, Xincheng Information Retrieval Artificial Intelligence Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains. |
| title | Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2601.11124 |