CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915920620814336 |
|---|---|
| author | Adam, Tim Lukas Konrad, Phongsakon Mark Terrenzi, Riccardo Lukas, Florian Girardo Yilmaz, Rahime Sierszecki, Krzysztof Ayvaz, Serkan |
| author_facet | Adam, Tim Lukas Konrad, Phongsakon Mark Terrenzi, Riccardo Lukas, Florian Girardo Yilmaz, Rahime Sierszecki, Krzysztof Ayvaz, Serkan |
| contents | In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2\%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05755 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models Adam, Tim Lukas Konrad, Phongsakon Mark Terrenzi, Riccardo Lukas, Florian Girardo Yilmaz, Rahime Sierszecki, Krzysztof Ayvaz, Serkan Software Engineering Artificial Intelligence In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2\%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs. |
| title | CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2604.05755 |