Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916295460519936 |
|---|---|
| author | Pasch, Stefan Petridis, Dimitirios Cutura, Jannic |
| author_facet | Pasch, Stefan Petridis, Dimitirios Cutura, Jannic |
| contents | This paper addresses the deduplication of multilingual textual data using advanced NLP tools. We compare a two-step method involving translation to English followed by embedding with mpnet, and a multilingual embedding model (distiluse). The two-step approach achieved a higher F1 score (82% vs. 60%), particularly with less widely used languages, which can be increased up to 89% by leveraging expert rules based on domain knowledge. We also highlight limitations related to token length constraints and computational efficiency. Our methodology suggests improvements for future multilingual deduplication tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13695 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models Pasch, Stefan Petridis, Dimitirios Cutura, Jannic Artificial Intelligence I.2.7 This paper addresses the deduplication of multilingual textual data using advanced NLP tools. We compare a two-step method involving translation to English followed by embedding with mpnet, and a multilingual embedding model (distiluse). The two-step approach achieved a higher F1 score (82% vs. 60%), particularly with less widely used languages, which can be increased up to 89% by leveraging expert rules based on domain knowledge. We also highlight limitations related to token length constraints and computational efficiency. Our methodology suggests improvements for future multilingual deduplication tasks. |
| title | Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models |
| topic | Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2406.13695 |