DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909635200417792 |
|---|---|
| author | Ma, Xueguang Lin, Xi Victoria Oguz, Barlas Lin, Jimmy Yih, Wen-tau Chen, Xilun |
| author_facet | Ma, Xueguang Lin, Xi Victoria Oguz, Barlas Lin, Jimmy Yih, Wen-tau Chen, Xilun |
| contents | Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18460 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers Ma, Xueguang Lin, Xi Victoria Oguz, Barlas Lin, Jimmy Yih, Wen-tau Chen, Xilun Computation and Language Information Retrieval Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization. |
| title | DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2502.18460 |