Zhao, Xingyu 
ORCID: 0000-0002-3474-349X, Huang, Wei, Banks, Alec, Cox, Victoria, Flynn, David, Schewe, Sven ORCID: 0000-0002-9093-9518 and Huang, Xiaowei ORCID: 0000-0001-6267-0366
(2021)
Assessing the Reliability of Deep Learning Classifiers Through
Robustness Evaluation and Operational Profiles.
In: AISafety 2021 Workshop co-located with IJCAI-21, 2021-8-21 - 2021-8-23, virtual.
|
Text
main_aisafety_workshop.pdf - Author Accepted Manuscript Download (1MB) | Preview |
Abstract
The utilisation of Deep Learning (DL) is advancing into increasingly more sophisticated applications. While it shows great potential to provide transformational capabilities, DL also raises new challenges regarding its reliability in critical functions. In this paper, we present a model-agnostic reliability assessment method for DL classifiers, based on evidence from robustness evaluation and the operational profile (OP) of a given application. We partition the input space into small cells and then "assemble" their robustness (to the ground truth) according to the OP, where estimators on the cells' robustness and OPs are provided. Reliability estimates in terms of the probability of misclassification per input (pmi) can be derived together with confidence levels. A prototype tool is demonstrated with simplified case studies. Model assumptions and extension to real-world applications are also discussed. While our model easily uncovers the inherent difficulties of assessing the DL dependability (e.g. lack of data with ground truth and scalability issues), we provide preliminary/compromised solutions to advance in this research direction.
| Item Type: | Conference or Workshop Item (Unspecified) |
|---|---|
| Additional Information: | Accepted by the AISafety'21 Workshop at IJCAI-21. To appear in a volume of CEUR Workshop Proceedings |
| Uncontrolled Keywords: | cs.LG, cs.LG |
| Divisions: | Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 03 Jun 2021 15:06 |
| Last Modified: | 18 Jan 2023 22:36 |
| Related URLs: | |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3124915 |
Share
Share
