Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.20523 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913054907695104 |
|---|---|
| author | Le, Viet-Man Tran, Thi Ngoc Trang Lubos, Sebastian Felfernig, Alexander Garber, Damian |
| author_facet | Le, Viet-Man Tran, Thi Ngoc Trang Lubos, Sebastian Felfernig, Alexander Garber, Damian |
| contents | We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 standard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88-89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy-cost trade-offs that inform model selection. These findings position LLMs as lightweight assistants for early variability validation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_20523 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis Le, Viet-Man Tran, Thi Ngoc Trang Lubos, Sebastian Felfernig, Alexander Garber, Damian Software Engineering Artificial Intelligence We study whether Large Language Models (LLMs) can perform feature model analysis operations (AOs) directly on semi-formal textual blueprints, i.e., concise constrained-language descriptions of feature hierarchies and constraints, enabling early validation in Software Product Line scoping. Using 12 state-of-the-art LLMs and 16 standard AOs, we compare their outputs against the solver-based oracle FLAMA. Results show that reasoning-optimized models (e.g., Grok 4 Fast Reasoning, Gemini 2.5 Pro) achieve 88-89% average accuracy across all evaluated blueprints and operations, approaching solver correctness. We identify systematic errors in structural parsing and constraint reasoning, and highlight accuracy-cost trade-offs that inform model selection. These findings position LLMs as lightweight assistants for early variability validation. |
| title | Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2604.20523 |