Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916400538320896 |
|---|---|
| author | Chen, Alvin Po-Chun Groshan, Ray von Bayern, Sean |
| author_facet | Chen, Alvin Po-Chun Groshan, Ray von Bayern, Sean |
| contents | Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability to significantly improve model performance by optimizing prompts and evaluate the input dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02517 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization Chen, Alvin Po-Chun Groshan, Ray von Bayern, Sean Computation and Language Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability to significantly improve model performance by optimizing prompts and evaluate the input dataset. |
| title | Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.02517 |