Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912900246929408 |
|---|---|
| author | Zhang, Weixing Jiang, Bowen Fu, Yuhong Koziolek, Anne Hebig, Regina Strüber, Daniel |
| author_facet | Zhang, Weixing Jiang, Bowen Fu, Yuhong Koziolek, Anne Hebig, Regina Strüber, Daniel |
| contents | Software languages evolve over time for reasons such as feature additions. When grammars evolve, textual instances that originally conformed to them may become outdated. While model-driven engineering provides many techniques for co-evolving models with metamodel changes, these approaches are not designed for textual DSLs and may lose human-relevant information such as layout and comments. This study systematically evaluates the potential of large language models (LLMs) for co-evolving grammars and instances of textual DSLs. Using Claude Sonnet 4.5 and GPT-5.2 across ten case languages with ten runs each, we assess both correctness and preservation of human-oriented information. Results show strong performance on small-scale cases ($\geq$94% precision and recall for instances requiring fewer than 20 modified lines), but performance degraded with scale: Claude maintains 85% recall at 40 lines, while GPT fails on the largest instances. Response time increases substantially with instance size, and grammar evolution complexity and deletion granularity affect performance more than change type. These findings clarify when LLM-based co-evolution is effective and where current limitations remain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11904 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation Zhang, Weixing Jiang, Bowen Fu, Yuhong Koziolek, Anne Hebig, Regina Strüber, Daniel Software Engineering Artificial Intelligence Software languages evolve over time for reasons such as feature additions. When grammars evolve, textual instances that originally conformed to them may become outdated. While model-driven engineering provides many techniques for co-evolving models with metamodel changes, these approaches are not designed for textual DSLs and may lose human-relevant information such as layout and comments. This study systematically evaluates the potential of large language models (LLMs) for co-evolving grammars and instances of textual DSLs. Using Claude Sonnet 4.5 and GPT-5.2 across ten case languages with ten runs each, we assess both correctness and preservation of human-oriented information. Results show strong performance on small-scale cases ($\geq$94% precision and recall for instances requiring fewer than 20 modified lines), but performance degraded with scale: Claude maintains 85% recall at 40 lines, while GPT fails on the largest instances. Response time increases substantially with instance size, and grammar evolution complexity and deletion granularity affect performance more than change type. These findings clarify when LLM-based co-evolution is effective and where current limitations remain. |
| title | Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2602.11904 |