Dependability analysis of deep reinforcement learning based robotics and autonomous systems through probabilistic model checking
Dependability analysis of deep reinforcement learning based robotics and autonomous systems through probabilistic model checking
While Deep Reinforcement Learning (DRL) provides transformational capabilities to the control of Robotics and Autonomous Systems (RAS), the black-box nature of DRL and uncertain deployment environments of RAS pose new challenges on its dependability. Although existing works impose constraints on the DRL policy to ensure successful completion of the mission, it is far from adequate to assess the DRL-driven RAS in a holistic way considering all dependability properties. In this paper, we formally define a set of dependability properties in temporal logic and construct a Discrete-Time Markov Chain (DTMC) to model the dynamics of risk/failures of a DRL-driven RAS interacting with the stochastic environment. We then conduct Probabilistic Model Checking (PMC) on the designed DTMC to verify those properties. Our experimental results show that the proposed method is effective as a holistic assessment framework while uncovering conflicts between the properties that may need trade-offs in training. Moreover, we find that the standard DRL training cannot improve dependability properties, thus requiring bespoke optimisation objectives. Finally, our method offers sensitivity analysis of dependability properties to disturbance levels from environments, providing insights for the assurance of real RAS.
5171-5178
Dong, Yi
355a62d9-5d1a-4c14-a900-9911e8c62453
Zhao, Xingyu
56d69104-77e5-4741-bca1-c0fa13f433fe
Huang, Xiaowei
ea80b217-6df4-4708-970d-93303f2a17e5
Dong, Yi
355a62d9-5d1a-4c14-a900-9911e8c62453
Zhao, Xingyu
56d69104-77e5-4741-bca1-c0fa13f433fe
Huang, Xiaowei
ea80b217-6df4-4708-970d-93303f2a17e5
Dong, Yi, Zhao, Xingyu and Huang, Xiaowei
(2022)
Dependability analysis of deep reinforcement learning based robotics and autonomous systems through probabilistic model checking.
In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022.
vol. 2022-October,
IEEE.
.
(doi:10.1109/IROS47612.2022.9981794).
Record type:
Conference or Workshop Item
(Paper)
Abstract
While Deep Reinforcement Learning (DRL) provides transformational capabilities to the control of Robotics and Autonomous Systems (RAS), the black-box nature of DRL and uncertain deployment environments of RAS pose new challenges on its dependability. Although existing works impose constraints on the DRL policy to ensure successful completion of the mission, it is far from adequate to assess the DRL-driven RAS in a holistic way considering all dependability properties. In this paper, we formally define a set of dependability properties in temporal logic and construct a Discrete-Time Markov Chain (DTMC) to model the dynamics of risk/failures of a DRL-driven RAS interacting with the stochastic environment. We then conduct Probabilistic Model Checking (PMC) on the designed DTMC to verify those properties. Our experimental results show that the proposed method is effective as a holistic assessment framework while uncovering conflicts between the properties that may need trade-offs in training. Moreover, we find that the standard DRL training cannot improve dependability properties, thus requiring bespoke optimisation objectives. Finally, our method offers sensitivity analysis of dependability properties to disturbance levels from environments, providing insights for the assurance of real RAS.
Text
Dependability Analysis of Deep Reinforcement Learning based Robotics and Autonomous Systems through Probabilistic Model Checking
- Accepted Manuscript
Restricted to Repository staff only until 26 December 2024.
Available under License Other.
Request a copy
More information
e-pub ahead of print date: 26 December 2022
Additional Information:
Funding Information:
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 956123. This work is supported by the UK Dstl through the project of Safety Argument for Learning-enabled Autonomous Underwater robots and the UK EPSRC through End-to-End Conceptual Guarding of Neural Architectures [EP/T026995/1]. XZ’s contribution is partially supported through Fellowships at the Assuring Autonomy International Programme.
Venue - Dates:
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, , Kyoto, Japan, 2022-10-23 - 2022-10-27
Identifiers
Local EPrints ID: 484270
URI: http://eprints.soton.ac.uk/id/eprint/484270
ISSN: 2153-0858
PURE UUID: f7aeacd9-3c1f-4f66-bf4c-19cf2a826a1b
Catalogue record
Date deposited: 13 Nov 2023 18:54
Last modified: 18 Mar 2024 04:17
Export record
Altmetrics
Contributors
Author:
Yi Dong
Author:
Xingyu Zhao
Author:
Xiaowei Huang
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics