Can Cross Encoders Produce Useful Sentence Embeddings?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916600179851264 |
|---|---|
| author | Ananthakrishnan, Haritha Dolby, Julian Kokel, Harsha Samulowitz, Horst Srinivas, Kavitha |
| author_facet | Ananthakrishnan, Haritha Dolby, Julian Kokel, Harsha Samulowitz, Horst Srinivas, Kavitha |
| contents | Cross encoders (CEs) are trained with sentence pairs to detect relatedness. As CEs require sentence pairs at inference, the prevailing view is that they can only be used as re-rankers in information retrieval pipelines. Dual encoders (DEs) are instead used to embed sentences, where sentence pairs are encoded by two separate encoders with shared weights at training, and a loss function that ensures the pair's embeddings lie close in vector space if the sentences are related. DEs however, require much larger datasets to train, and are less accurate than CEs. We report a curious finding that embeddings from earlier layers of CEs can in fact be used within an information retrieval pipeline. We show how to exploit CEs to distill a lighter-weight DE, with a 5.15x speedup in inference time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03552 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Cross Encoders Produce Useful Sentence Embeddings? Ananthakrishnan, Haritha Dolby, Julian Kokel, Harsha Samulowitz, Horst Srinivas, Kavitha Computation and Language Information Retrieval Cross encoders (CEs) are trained with sentence pairs to detect relatedness. As CEs require sentence pairs at inference, the prevailing view is that they can only be used as re-rankers in information retrieval pipelines. Dual encoders (DEs) are instead used to embed sentences, where sentence pairs are encoded by two separate encoders with shared weights at training, and a loss function that ensures the pair's embeddings lie close in vector space if the sentences are related. DEs however, require much larger datasets to train, and are less accurate than CEs. We report a curious finding that embeddings from earlier layers of CEs can in fact be used within an information retrieval pipeline. We show how to exploit CEs to distill a lighter-weight DE, with a 5.15x speedup in inference time. |
| title | Can Cross Encoders Produce Useful Sentence Embeddings? |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2502.03552 |