Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913104096395264 |
|---|---|
| author | Dillon, Drew Varanasi, Kasyap |
| author_facet | Dillon, Drew Varanasi, Kasyap |
| contents | AI coding agents powered by large language models can read codebases and produce functional code, but they routinely violate team-specific product decisions that are invisible in the source code alone. We introduce a controlled benchmark measuring decision compliance, the rate at which an AI coding agent follows established product, design, and engineering decisions, across 8 realistic software engineering tasks containing 41 weighted decision points. We compare a baseline configuration (Claude Code with codebase access only) against an augmented configuration that adds Brief, a product-context retrieval system providing spec generation, mid-build consultation, and retrieval of recorded decisions, persona pain points, customer signals, and competitive intelligence. On identical prompts and the same repository, the augmented configuration achieves 95% decision compliance versus 46% for the baseline, a 49 percentage point improvement. Per-decision analysis reveals that the baseline achieves 100% compliance on decisions visible in the codebase and 0-33% on decisions requiring product context, suggesting that product-context retrieval is a key driver of the improvement. We release the benchmark repository, all 16 pull requests, and scoring harness for independent reproduction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08112 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49% Dillon, Drew Varanasi, Kasyap Software Engineering Artificial Intelligence Computational Engineering, Finance, and Science Machine Learning Logic in Computer Science D.2.1; I.2.2; D.2.5 AI coding agents powered by large language models can read codebases and produce functional code, but they routinely violate team-specific product decisions that are invisible in the source code alone. We introduce a controlled benchmark measuring decision compliance, the rate at which an AI coding agent follows established product, design, and engineering decisions, across 8 realistic software engineering tasks containing 41 weighted decision points. We compare a baseline configuration (Claude Code with codebase access only) against an augmented configuration that adds Brief, a product-context retrieval system providing spec generation, mid-build consultation, and retrieval of recorded decisions, persona pain points, customer signals, and competitive intelligence. On identical prompts and the same repository, the augmented configuration achieves 95% decision compliance versus 46% for the baseline, a 49 percentage point improvement. Per-decision analysis reveals that the baseline achieves 100% compliance on decisions visible in the codebase and 0-33% on decisions requiring product context, suggesting that product-context retrieval is a key driver of the improvement. We release the benchmark repository, all 16 pull requests, and scoring harness for independent reproduction. |
| title | Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49% |
| topic | Software Engineering Artificial Intelligence Computational Engineering, Finance, and Science Machine Learning Logic in Computer Science D.2.1; I.2.2; D.2.5 |
| url | https://arxiv.org/abs/2605.08112 |