Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.16031 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909150386061312 |
|---|---|
| author | Saggau, Daniel Rezaei, Mina Bischl, Bernd Chalkidis, Ilias |
| author_facet | Saggau, Daniel Rezaei, Mina Bischl, Bernd Chalkidis, Ilias |
| contents | Learning quality document embeddings is a fundamental problem in natural language processing (NLP), information retrieval (IR), recommendation systems, and search engines. Despite recent advances in the development of transformer-based models that produce sentence embeddings with self-contrastive learning, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations. Therefore, we train Longfomer-based document encoders using a state-of-the-art unsupervised contrastive learning method (SimCSE). Further on, we complement the baseline method -- siamese neural network -- with additional convex neural networks based on functional Bregman divergence aiming to enhance the quality of the output document representations. We show that overall the combination of a self-contrastive siamese network and our proposed neural Bregman network outperforms the baselines in two linear classification settings on three long document topic classification tasks from the legal and biomedical domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_16031 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning Saggau, Daniel Rezaei, Mina Bischl, Bernd Chalkidis, Ilias Computation and Language Learning quality document embeddings is a fundamental problem in natural language processing (NLP), information retrieval (IR), recommendation systems, and search engines. Despite recent advances in the development of transformer-based models that produce sentence embeddings with self-contrastive learning, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations. Therefore, we train Longfomer-based document encoders using a state-of-the-art unsupervised contrastive learning method (SimCSE). Further on, we complement the baseline method -- siamese neural network -- with additional convex neural networks based on functional Bregman divergence aiming to enhance the quality of the output document representations. We show that overall the combination of a self-contrastive siamese network and our proposed neural Bregman network outperforms the baselines in two linear classification settings on three long document topic classification tasks from the legal and biomedical domains. |
| title | Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2305.16031 |