LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911505562206208 |
|---|---|
| author | Yang, Hsin-Jung Jiang, Zhanhong Koirala, Prajwal Liu, Qisai Fleming, Cody Sarkar, Soumik |
| author_facet | Yang, Hsin-Jung Jiang, Zhanhong Koirala, Prajwal Liu, Qisai Fleming, Cody Sarkar, Soumik |
| contents | Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17312 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy Yang, Hsin-Jung Jiang, Zhanhong Koirala, Prajwal Liu, Qisai Fleming, Cody Sarkar, Soumik Machine Learning Systems and Control Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making. |
| title | LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy |
| topic | Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2602.17312 |