Executing Natural Language-Described Algorithms with Large Language Models: An Investigation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909136210362368 |
|---|---|
| author | Zheng, Xin Zhu, Qiming Lin, Hongyu Lu, Yaojie Han, Xianpei Sun, Le |
| author_facet | Zheng, Xin Zhu, Qiming Lin, Hongyu Lu, Yaojie Han, Xianpei Sun, Le |
| contents | Executing computer programs described in natural language has long been a pursuit of computer science. With the advent of enhanced natural language understanding capabilities exhibited by large language models (LLMs), the path toward this goal has been illuminated. In this paper, we seek to examine the capacity of present-day LLMs to comprehend and execute algorithms outlined in natural language. We established an algorithm test set sourced from Introduction to Algorithm, a well-known textbook that contains many representative widely-used algorithms. To systematically assess LLMs' code execution abilities, we selected 30 algorithms, generated 300 random-sampled instances in total, and evaluated whether popular LLMs can understand and execute these algorithms. Our findings reveal that LLMs, notably GPT-4, can effectively execute programs described in natural language, as long as no heavy numeric computation is involved. We believe our findings contribute to evaluating LLMs' code execution abilities and would encourage further investigation and application for the computation power of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00795 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Executing Natural Language-Described Algorithms with Large Language Models: An Investigation Zheng, Xin Zhu, Qiming Lin, Hongyu Lu, Yaojie Han, Xianpei Sun, Le Computation and Language Artificial Intelligence Executing computer programs described in natural language has long been a pursuit of computer science. With the advent of enhanced natural language understanding capabilities exhibited by large language models (LLMs), the path toward this goal has been illuminated. In this paper, we seek to examine the capacity of present-day LLMs to comprehend and execute algorithms outlined in natural language. We established an algorithm test set sourced from Introduction to Algorithm, a well-known textbook that contains many representative widely-used algorithms. To systematically assess LLMs' code execution abilities, we selected 30 algorithms, generated 300 random-sampled instances in total, and evaluated whether popular LLMs can understand and execute these algorithms. Our findings reveal that LLMs, notably GPT-4, can effectively execute programs described in natural language, as long as no heavy numeric computation is involved. We believe our findings contribute to evaluating LLMs' code execution abilities and would encourage further investigation and application for the computation power of LLMs. |
| title | Executing Natural Language-Described Algorithms with Large Language Models: An Investigation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.00795 |