iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917674854907904 |
|---|---|
| author | Gupta, Harshit Chaudhary, Manav Raha, Tathagata Subramanian, Shivansh Varma, Vasudeva |
| author_facet | Gupta, Harshit Chaudhary, Manav Raha, Tathagata Subramanian, Shivansh Varma, Vasudeva |
| contents | This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models' lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default common-sense associations and exhibit unconventional thinking. We propose a unique strategy to improve the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dynamic few-shot prompting techniques and introduce a model-generated reasoning strategy that utilizes the LLM's reasoning capabilities to improve performance. Our approach demonstrated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of performing as well as the human annotators, thus highlighting the efficacy of the proposed strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16129 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers Gupta, Harshit Chaudhary, Manav Raha, Tathagata Subramanian, Shivansh Varma, Vasudeva Computation and Language This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models' lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default common-sense associations and exhibit unconventional thinking. We propose a unique strategy to improve the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dynamic few-shot prompting techniques and introduce a model-generated reasoning strategy that utilizes the LLM's reasoning capabilities to improve performance. Our approach demonstrated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of performing as well as the human annotators, thus highlighting the efficacy of the proposed strategies. |
| title | iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.16129 |