mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914971180335104 |
|---|---|
| author | Zhang, Xin Zhang, Yanzhao Long, Dingkun Xie, Wen Dai, Ziqi Tang, Jialong Lin, Huan Yang, Baosong Xie, Pengjun Huang, Fei Zhang, Meishan Li, Wenjie Zhang, Min |
| author_facet | Zhang, Xin Zhang, Yanzhao Long, Dingkun Xie, Wen Dai, Ziqi Tang, Jialong Lin, Huan Yang, Baosong Xie, Pengjun Huang, Fei Zhang, Meishan Li, Wenjie Zhang, Min |
| contents | We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19669 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval Zhang, Xin Zhang, Yanzhao Long, Dingkun Xie, Wen Dai, Ziqi Tang, Jialong Lin, Huan Yang, Baosong Xie, Pengjun Huang, Fei Zhang, Meishan Li, Wenjie Zhang, Min Computation and Language Information Retrieval We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications. |
| title | mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2407.19669 |