Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918397015490560 |
|---|---|
| author | Liu, Shiyan Xia, Qifeng Xia, Qiyun Liu, Yisheng Yu, Xinyu Qu, Rui |
| author_facet | Liu, Shiyan Xia, Qifeng Xia, Qiyun Liu, Yisheng Yu, Xinyu Qu, Rui |
| contents | Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_18388 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization Liu, Shiyan Xia, Qifeng Xia, Qiyun Liu, Yisheng Yu, Xinyu Qu, Rui Artificial Intelligence Multiagent Systems Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025. |
| title | Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization |
| topic | Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2603.18388 |