A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917465645121536 |
|---|---|
| author | Barkley, Brett Fridovich-Keil, David |
| author_facet | Barkley, Brett Fridovich-Keil, David |
| contents | Synthetic data is central to data-efficient Dyna-style model-based reinforcement learning, but it can also degrade performance. We study this failure in Model-Based Policy Optimization (MBPO), which performs actor-critic updates using model-generated synthetic state transitions. Although MBPO reports strong sample-efficiency gains on OpenAI Gym, recent work shows that it often underperforms Soft Actor-Critic (SAC), its non-Dyna base, in the DeepMind Control Suite (DMC), despite both suites involving MuJoCo-based proprioceptive continuous control. We identify two coupled causes of this collapse: scale mismatch between dynamics and reward targets, which suppresses reward learning and induces critic underestimation, and residual next-state prediction, which inflates model variance and produces unreliable synthetic transitions. We introduce Fixing That Free Lunch (FTFL), a minimal repair that combines independent target normalization with direct next-state prediction. FTFL outperforms SAC in five of seven previously failing DMC tasks while preserving MBPO's strong Gym performance. We further show that MBPO-lineage algorithms, including uncertainty-aware variants that filter, penalize, or reject synthetic transitions based on model uncertainty, still inherit these failures unless FTFL is applied to their shared learned-model backbone. More broadly, our results show how benchmark-limited evaluation can encode environment-specific assumptions into algorithm design, motivating taxonomies that map MDP structure to algorithmic failure modes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimization Barkley, Brett Fridovich-Keil, David Machine Learning Synthetic data is central to data-efficient Dyna-style model-based reinforcement learning, but it can also degrade performance. We study this failure in Model-Based Policy Optimization (MBPO), which performs actor-critic updates using model-generated synthetic state transitions. Although MBPO reports strong sample-efficiency gains on OpenAI Gym, recent work shows that it often underperforms Soft Actor-Critic (SAC), its non-Dyna base, in the DeepMind Control Suite (DMC), despite both suites involving MuJoCo-based proprioceptive continuous control. We identify two coupled causes of this collapse: scale mismatch between dynamics and reward targets, which suppresses reward learning and induces critic underestimation, and residual next-state prediction, which inflates model variance and produces unreliable synthetic transitions. We introduce Fixing That Free Lunch (FTFL), a minimal repair that combines independent target normalization with direct next-state prediction. FTFL outperforms SAC in five of seven previously failing DMC tasks while preserving MBPO's strong Gym performance. We further show that MBPO-lineage algorithms, including uncertainty-aware variants that filter, penalize, or reject synthetic transitions based on model uncertainty, still inherit these failures unless FTFL is applied to their shared learned-model backbone. More broadly, our results show how benchmark-limited evaluation can encode environment-specific assumptions into algorithm design, motivating taxonomies that map MDP structure to algorithmic failure modes. |
| title | A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.01457 |