Wang, Y, Zhan, SS, Jiao, R, Wang, Z, Jin, W, Yang, Z, Wang, Z, Huang, C 
ORCID: 0000-0002-9300-1787 and Zhu, Q
(2023)
Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments.
In: International Conference on Machine Learning, 2023-7-23 - 2023-7-29, Hawaii, US.
|
Text
icml2023_camera_ready.pdf - Author Accepted Manuscript Available under License Creative Commons Attribution. Download (740kB) | Preview |
Abstract
Reinforcement Learning (RL) has long grappled with the issue of ensuring agent safety in unpredictable and stochastic environments, particularly under hard constraints that require the system state not to reach unsafe regions. Conventional safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in a cost function and try to constrain the expectation of cumulative cost under a threshold. However, it is often difficult to effectively capture and enforce hard reachability-based safety constraints indirectly with such constraints on safety violation cost. In this work, we leverage the notion of barrier function to explicitly encode the hard safety chance constraints, and as the environment is unknown, relax them to our design of generative-model-based soft barrier functions. Based on such soft barriers, we propose a novel safe RL approach with bi-level optimization that can jointly learn the unknown environment and optimize the control policy, while effectively avoiding the unsafe region with safety probability optimization. Experiments on a set of examples demonstrate that our approach can effectively enforce hard safety chance constraints and significantly outperform CMDP-based baseline methods in system safe rates measured via simulations.
| Item Type: | Conference or Workshop Item (Unspecified) |
|---|---|
| Divisions: | Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 03 Jul 2023 07:58 |
| Last Modified: | 22 Nov 2023 14:10 |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3171396 |
Share
Share
