Succeeding at Scale: Automated Dataset Construction and Query-Side Adaptation for Multi-Tenant Search
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908864555778048 |
|---|---|
| author | Jain, Prateek Nair, Shabari S Goru, Ritesh Agarwal, Prakhar Yadav, Ajay Varadharajan, Yoga Sri Varshan Caramanis, Constantine |
| author_facet | Jain, Prateek Nair, Shabari S Goru, Ritesh Agarwal, Prakhar Yadav, Ajay Varadharajan, Yoga Sri Varshan Caramanis, Constantine |
| contents | Large-scale multi-tenant retrieval systems generate extensive query logs but lack curated relevance labels for effective domain adaptation, resulting in substantial underutilized "dark data". This challenge is compounded by the high cost of model updates, as jointly fine-tuning query and document encoders requires full corpus re-indexing, which is impractical in multi-tenant settings with thousands of isolated indices. We introduce DevRev-Search, a passage retrieval benchmark for technical customer support built via a fully automated pipeline. Candidate generation uses fusion across diverse sparse and dense retrievers, followed by an LLM-as-a-Judge for consistency filtering and relevance labeling. We further propose an Index-Preserving Adaptation strategy that fine-tunes only the query encoder, achieving strong performance gains while keeping document indices fixed. Experiments on DevRev-Search, SciFact, and FiQA-2018 show that Parameter-Efficient Fine-Tuning (PEFT) of the query encoder delivers a remarkable quality-efficiency trade-off, enabling scalable and practical enterprise search adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04646 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Succeeding at Scale: Automated Dataset Construction and Query-Side Adaptation for Multi-Tenant Search Jain, Prateek Nair, Shabari S Goru, Ritesh Agarwal, Prakhar Yadav, Ajay Varadharajan, Yoga Sri Varshan Caramanis, Constantine Information Retrieval Artificial Intelligence Computation and Language Machine Learning Large-scale multi-tenant retrieval systems generate extensive query logs but lack curated relevance labels for effective domain adaptation, resulting in substantial underutilized "dark data". This challenge is compounded by the high cost of model updates, as jointly fine-tuning query and document encoders requires full corpus re-indexing, which is impractical in multi-tenant settings with thousands of isolated indices. We introduce DevRev-Search, a passage retrieval benchmark for technical customer support built via a fully automated pipeline. Candidate generation uses fusion across diverse sparse and dense retrievers, followed by an LLM-as-a-Judge for consistency filtering and relevance labeling. We further propose an Index-Preserving Adaptation strategy that fine-tunes only the query encoder, achieving strong performance gains while keeping document indices fixed. Experiments on DevRev-Search, SciFact, and FiQA-2018 show that Parameter-Efficient Fine-Tuning (PEFT) of the query encoder delivers a remarkable quality-efficiency trade-off, enabling scalable and practical enterprise search adaptation. |
| title | Succeeding at Scale: Automated Dataset Construction and Query-Side Adaptation for Multi-Tenant Search |
| topic | Information Retrieval Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.04646 |