A Relative Ignorability Framework for Decision-Relevant Observability in Control Theory and Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911094157606912 |
|---|---|
| author | Bleile, MaryLena Phung, Minh-Nhat Tran, Minh-Binh |
| author_facet | Bleile, MaryLena Phung, Minh-Nhat Tran, Minh-Binh |
| contents | Sequential decision-making systems routinely operate with missing or incomplete data. Classical reinforcement learning theory, which is commonly used to solve sequential decision problems, assumes Markovian observability, which may not hold under partial observability. Causal inference paradigms formalise ignorability of missingness. We show these views can be unified and generalized in order to guarantee Q-learning convergence even when the Markov property fails. To do so, we introduce the concept of relative ignorability. Relative ignorability is a graphical-causal criterion which refines the requirements for accurate decision-making based on incomplete data. Theoretical results and simulations both reveal that non-Markovian stochastic processes whose missingness is relatively ignorable with respect to causal estimands can still be optimized using standard Reinforcement Learning algorithms. These results expand the theoretical foundations of safe, data-efficient AI to real-world environments where complete information is unattainable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07722 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Relative Ignorability Framework for Decision-Relevant Observability in Control Theory and Reinforcement Learning Bleile, MaryLena Phung, Minh-Nhat Tran, Minh-Binh Machine Learning Methodology 60G Sequential decision-making systems routinely operate with missing or incomplete data. Classical reinforcement learning theory, which is commonly used to solve sequential decision problems, assumes Markovian observability, which may not hold under partial observability. Causal inference paradigms formalise ignorability of missingness. We show these views can be unified and generalized in order to guarantee Q-learning convergence even when the Markov property fails. To do so, we introduce the concept of relative ignorability. Relative ignorability is a graphical-causal criterion which refines the requirements for accurate decision-making based on incomplete data. Theoretical results and simulations both reveal that non-Markovian stochastic processes whose missingness is relatively ignorable with respect to causal estimands can still be optimized using standard Reinforcement Learning algorithms. These results expand the theoretical foundations of safe, data-efficient AI to real-world environments where complete information is unattainable. |
| title | A Relative Ignorability Framework for Decision-Relevant Observability in Control Theory and Reinforcement Learning |
| topic | Machine Learning Methodology 60G |
| url | https://arxiv.org/abs/2504.07722 |