Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrieval
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913875738230784 |
|---|---|
| author | Roychowdhury, Sujoy Soman, Sumit Gireesha, Ranjani Hosakere Chhabra, Vansh Gunda, Neeraj Bandyopadhyay, Subhadip Bala, Sai Krishna |
| author_facet | Roychowdhury, Sujoy Soman, Sumit Gireesha, Ranjani Hosakere Chhabra, Vansh Gunda, Neeraj Bandyopadhyay, Subhadip Bala, Sai Krishna |
| contents | A plethora of sentence embedding models makes it challenging to choose one, especially for technical domains rich with specialized vocabulary. In this work, we domain adapt embeddings using telecom data for question answering. We evaluate embeddings obtained from publicly available models and their domain-adapted variants, on both point retrieval accuracies, as well as their (95%) confidence intervals. We establish a systematic method to obtain thresholds for similarity scores for different embeddings. As expected, we observe that fine-tuning improves mean bootstrapped accuracies. We also observe that it results in tighter confidence intervals, which further improve when pre-training is preceded by fine-tuning. We introduce metrics which measure the distributional overlaps of top-$K$, correct and random document similarities with the question. Further, we show that these metrics are correlated with retrieval accuracy and similarity thresholds. Recent literature shows conflicting effects of isotropy on retrieval accuracies. Our experiments establish that the isotropy of embeddings (as measured by two independent state-of-the-art isotropy metric definitions) is poorly correlated with retrieval performance. We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones, and fine-tuning moves them further apart. Based on our results, we provide recommendations for use of our methodology and metrics by researchers and practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12336 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrieval Roychowdhury, Sujoy Soman, Sumit Gireesha, Ranjani Hosakere Chhabra, Vansh Gunda, Neeraj Bandyopadhyay, Subhadip Bala, Sai Krishna Computation and Language Artificial Intelligence Machine Learning 68T50 I.2.7 A plethora of sentence embedding models makes it challenging to choose one, especially for technical domains rich with specialized vocabulary. In this work, we domain adapt embeddings using telecom data for question answering. We evaluate embeddings obtained from publicly available models and their domain-adapted variants, on both point retrieval accuracies, as well as their (95%) confidence intervals. We establish a systematic method to obtain thresholds for similarity scores for different embeddings. As expected, we observe that fine-tuning improves mean bootstrapped accuracies. We also observe that it results in tighter confidence intervals, which further improve when pre-training is preceded by fine-tuning. We introduce metrics which measure the distributional overlaps of top-$K$, correct and random document similarities with the question. Further, we show that these metrics are correlated with retrieval accuracy and similarity thresholds. Recent literature shows conflicting effects of isotropy on retrieval accuracies. Our experiments establish that the isotropy of embeddings (as measured by two independent state-of-the-art isotropy metric definitions) is poorly correlated with retrieval performance. We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones, and fine-tuning moves them further apart. Based on our results, we provide recommendations for use of our methodology and metrics by researchers and practitioners. |
| title | Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrieval |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T50 I.2.7 |
| url | https://arxiv.org/abs/2406.12336 |