Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914689692205056 |
|---|---|
| author | Liu, Junlong Shang, Xichen Feng, Huawen Zheng, Junhao Ma, Qianli |
| author_facet | Liu, Junlong Shang, Xichen Feng, Huawen Zheng, Junhao Ma, Qianli |
| contents | Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings Liu, Junlong Shang, Xichen Feng, Huawen Zheng, Junhao Ma, Qianli Computation and Language Machine Learning Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks. |
| title | Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.15153 |