Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.17644 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911119777464320 |
|---|---|
| author | Alaofi, Marwah Ferro, Nicola Thomas, Paul Scholer, Falk Sanderson, Mark |
| author_facet | Alaofi, Marwah Ferro, Nicola Thomas, Paul Scholer, Falk Sanderson, Mark |
| contents | This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an 'ideal' test collection. A Large Language Model (LLM) is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR literature to influence query formulation.
The LLM's ability to generate query variants that align with user profiles is empirically validated, and the variants' utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_17644 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Demographically-Inspired Query Variants Using an LLM Alaofi, Marwah Ferro, Nicola Thomas, Paul Scholer, Falk Sanderson, Mark Information Retrieval This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an 'ideal' test collection. A Large Language Model (LLM) is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR literature to influence query formulation. The LLM's ability to generate query variants that align with user profiles is empirically validated, and the variants' utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users. |
| title | Demographically-Inspired Query Variants Using an LLM |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2508.17644 |