RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917597855875072 |
|---|---|
| author | Zhang, Yuansen Wang, Xiao Xi, Zhiheng Xia, Han Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Zhang, Yuansen Wang, Xiao Xi, Zhiheng Xia, Han Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Large Language Models (LLMs) have showcased remarkable capabilities in following human instructions. However, recent studies have raised concerns about the robustness of LLMs when prompted with instructions combining textual adversarial samples. In this paper, drawing inspiration from recent works that LLMs are sensitive to the design of the instructions, we utilize instructions in code style, which are more structural and less ambiguous, to replace typically natural language instructions. Through this conversion, we provide LLMs with more precise instructions and strengthen the robustness of LLMs. Moreover, under few-shot scenarios, we propose a novel method to compose in-context demonstrations using both clean and adversarial samples (\textit{adversarial context method}) to further boost the robustness of the LLMs. Experiments on eight robustness datasets show that our method consistently outperforms prompting LLMs with natural language instructions. For example, with gpt-3.5-turbo, our method achieves an improvement of 5.68\% in test set accuracy and a reduction of 5.66 points in Attack Success Rate (ASR). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16431 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions Zhang, Yuansen Wang, Xiao Xi, Zhiheng Xia, Han Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Large Language Models (LLMs) have showcased remarkable capabilities in following human instructions. However, recent studies have raised concerns about the robustness of LLMs when prompted with instructions combining textual adversarial samples. In this paper, drawing inspiration from recent works that LLMs are sensitive to the design of the instructions, we utilize instructions in code style, which are more structural and less ambiguous, to replace typically natural language instructions. Through this conversion, we provide LLMs with more precise instructions and strengthen the robustness of LLMs. Moreover, under few-shot scenarios, we propose a novel method to compose in-context demonstrations using both clean and adversarial samples (\textit{adversarial context method}) to further boost the robustness of the LLMs. Experiments on eight robustness datasets show that our method consistently outperforms prompting LLMs with natural language instructions. For example, with gpt-3.5-turbo, our method achieves an improvement of 5.68\% in test set accuracy and a reduction of 5.66 points in Attack Success Rate (ASR). |
| title | RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.16431 |