Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929249035747328 |
|---|---|
| author | Chen, Yuhao Wong, Chloe Yang, Hanwen Aguenza, Juan Bhujangari, Sai Vu, Benthan Lei, Xun Prasad, Amisha Fluss, Manny Phuong, Eric Liu, Minghao Kumar, Raja Vats, Vanshika Davis, James |
| author_facet | Chen, Yuhao Wong, Chloe Yang, Hanwen Aguenza, Juan Bhujangari, Sai Vu, Benthan Lei, Xun Prasad, Amisha Fluss, Manny Phuong, Eric Liu, Minghao Kumar, Raja Vats, Vanshika Davis, James |
| contents | This study critically evaluates the efficacy of prompting methods in enhancing the mathematical reasoning capability of large language models (LLMs). The investigation uses three prescriptive prompting methods - simple, persona, and conversational prompting - known for their effectiveness in enhancing the linguistic tasks of LLMs. We conduct this analysis on OpenAI's LLM chatbot, ChatGPT-3.5, on extensive problem sets from the MATH, GSM8K, and MMLU datasets, encompassing a broad spectrum of mathematical challenges. A grading script adapted to each dataset is used to determine the effectiveness of these prompting interventions in enhancing the model's mathematical analysis power. Contrary to expectations, our empirical analysis reveals that none of the investigated methods consistently improves over ChatGPT-3.5's baseline performance, with some causing significant degradation. Our findings suggest that prompting strategies do not necessarily generalize to new domains, in this study failing to enhance mathematical performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_15006 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities Chen, Yuhao Wong, Chloe Yang, Hanwen Aguenza, Juan Bhujangari, Sai Vu, Benthan Lei, Xun Prasad, Amisha Fluss, Manny Phuong, Eric Liu, Minghao Kumar, Raja Vats, Vanshika Davis, James Artificial Intelligence Computation and Language Machine Learning This study critically evaluates the efficacy of prompting methods in enhancing the mathematical reasoning capability of large language models (LLMs). The investigation uses three prescriptive prompting methods - simple, persona, and conversational prompting - known for their effectiveness in enhancing the linguistic tasks of LLMs. We conduct this analysis on OpenAI's LLM chatbot, ChatGPT-3.5, on extensive problem sets from the MATH, GSM8K, and MMLU datasets, encompassing a broad spectrum of mathematical challenges. A grading script adapted to each dataset is used to determine the effectiveness of these prompting interventions in enhancing the model's mathematical analysis power. Contrary to expectations, our empirical analysis reveals that none of the investigated methods consistently improves over ChatGPT-3.5's baseline performance, with some causing significant degradation. Our findings suggest that prompting strategies do not necessarily generalize to new domains, in this study failing to enhance mathematical performance. |
| title | Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2312.15006 |