ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908278647160832 |
|---|---|
| author | Kammakomati, Mehant Pimparkhede, Sameer Tamilselvam, Srikanth Kumar, Prince Bhattacharyya, Pushpak |
| author_facet | Kammakomati, Mehant Pimparkhede, Sameer Tamilselvam, Srikanth Kumar, Prince Bhattacharyya, Pushpak |
| contents | Recent work shows Large Language Models (LLMs) struggle to understand natural language constraints for various text generation tasks in zero- and few-shot settings. While, in the code domain, there is wide usage of constraints in code format to maintain the integrity of code written in Domain-Specific Languages (DSLs) like JSON and YAML which are widely used for system-level programming tasks in enterprises. Given that LLMs are increasingly used for system-level code tasks, evaluating if they can comprehend these code constraints is crucial. However, no work has been done to evaluate their controllability over code constraints. Hence, we introduce ConCodeEval, a first-of-its-kind benchmark having two novel tasks for code constraints across five representations. Our findings suggest that language models struggle with code constraints. Code languages that perform excellently for normal code tasks do not perform well when the same languages represent fine-grained constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03387 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages Kammakomati, Mehant Pimparkhede, Sameer Tamilselvam, Srikanth Kumar, Prince Bhattacharyya, Pushpak Software Engineering Artificial Intelligence Computation and Language Recent work shows Large Language Models (LLMs) struggle to understand natural language constraints for various text generation tasks in zero- and few-shot settings. While, in the code domain, there is wide usage of constraints in code format to maintain the integrity of code written in Domain-Specific Languages (DSLs) like JSON and YAML which are widely used for system-level programming tasks in enterprises. Given that LLMs are increasingly used for system-level code tasks, evaluating if they can comprehend these code constraints is crucial. However, no work has been done to evaluate their controllability over code constraints. Hence, we introduce ConCodeEval, a first-of-its-kind benchmark having two novel tasks for code constraints across five representations. Our findings suggest that language models struggle with code constraints. Code languages that perform excellently for normal code tasks do not perform well when the same languages represent fine-grained constraints. |
| title | ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.03387 |