Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929388204851200 |
|---|---|
| author | Xian, Jasper Samuel, Saron Khoubsirat, Faraz Pradeep, Ronak Sultan, Md Arafat Florian, Radu Roukos, Salim Sil, Avirup Potts, Christopher Khattab, Omar |
| author_facet | Xian, Jasper Samuel, Saron Khoubsirat, Faraz Pradeep, Ronak Sultan, Md Arafat Florian, Radu Roukos, Salim Sil, Avirup Potts, Christopher Khattab, Omar |
| contents | We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11706 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels Xian, Jasper Samuel, Saron Khoubsirat, Faraz Pradeep, Ronak Sultan, Md Arafat Florian, Radu Roukos, Salim Sil, Avirup Potts, Christopher Khattab, Omar Information Retrieval Computation and Language Machine Learning We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation. |
| title | Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels |
| topic | Information Retrieval Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.11706 |