APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910954193682432 |
|---|---|
| author | Jin, Can Peng, Hongwu Zhao, Shiyu Wang, Zhenting Xu, Wujiang Han, Ligong Zhao, Jiahui Zhong, Kai Rajasekaran, Sanguthevar Metaxas, Dimitris N. |
| author_facet | Jin, Can Peng, Hongwu Zhao, Shiyu Wang, Zhenting Xu, Wujiang Han, Ligong Zhao, Jiahui Zhong, Kai Rajasekaran, Sanguthevar Metaxas, Dimitris N. |
| contents | Large Language Models (LLMs) have significantly enhanced Information Retrieval (IR) across various modules, such as reranking. Despite impressive performance, current zero-shot relevance ranking with LLMs heavily relies on human prompt engineering. Existing automatic prompt engineering algorithms primarily focus on language modeling and classification tasks, leaving the domain of IR, particularly reranking, underexplored. Directly applying current prompt engineering algorithms to relevance ranking is challenging due to the integration of query and long passage pairs in the input, where the ranking complexity surpasses classification tasks. To reduce human effort and unlock the potential of prompt optimization in reranking, we introduce a novel automatic prompt engineering algorithm named APEER. APEER iteratively generates refined prompts through feedback and preference optimization. Extensive experiments with four LLMs and ten datasets demonstrate the substantial performance improvement of APEER over existing state-of-the-art (SoTA) manual prompts. Furthermore, we find that the prompts generated by APEER exhibit better transferability across diverse tasks and LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14449 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking Jin, Can Peng, Hongwu Zhao, Shiyu Wang, Zhenting Xu, Wujiang Han, Ligong Zhao, Jiahui Zhong, Kai Rajasekaran, Sanguthevar Metaxas, Dimitris N. Artificial Intelligence Large Language Models (LLMs) have significantly enhanced Information Retrieval (IR) across various modules, such as reranking. Despite impressive performance, current zero-shot relevance ranking with LLMs heavily relies on human prompt engineering. Existing automatic prompt engineering algorithms primarily focus on language modeling and classification tasks, leaving the domain of IR, particularly reranking, underexplored. Directly applying current prompt engineering algorithms to relevance ranking is challenging due to the integration of query and long passage pairs in the input, where the ranking complexity surpasses classification tasks. To reduce human effort and unlock the potential of prompt optimization in reranking, we introduce a novel automatic prompt engineering algorithm named APEER. APEER iteratively generates refined prompts through feedback and preference optimization. Extensive experiments with four LLMs and ten datasets demonstrate the substantial performance improvement of APEER over existing state-of-the-art (SoTA) manual prompts. Furthermore, we find that the prompts generated by APEER exhibit better transferability across diverse tasks and LLMs. |
| title | APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2406.14449 |