Hierarchical Distribution-aware Testing of Deep Learning

Huang, Wei ORCID: 0000-0003-1418-6267, Zhao, Xingyu ORCID: 0000-0002-3474-349X, Banks, Alec ORCID: 0000-0002-1350-0798, Cox, Victoria ORCID: 0000-0003-3852-7855 and Huang, Xiaowei ORCID: 0000-0001-6267-0366 (2024) Hierarchical Distribution-aware Testing of Deep Learning. ACM Transactions on Software Engineering and Methodology, 33 (2). pp. 1-35.

PDF
2205.08589.pdf - Author Accepted Manuscript
Download (3MB) | Preview

Official URL: http://dx.doi.org/10.1145/3625290

Abstract

<jats:p> With its growing use in safety/security-critical applications, Deep Learning (DL) has raised increasing concerns regarding its dependability. In particular, DL has a notorious problem of lacking robustness. Input added with adversarial perturbations, i.e., <jats:bold>Adversarial Examples (AEs)</jats:bold> , are easily mispredicted by the DL model. Despite recent efforts made in detecting AEs via state-of-the-art attack and testing methods, they are normally input distribution–agnostic and/or disregard the perceptual quality of adversarial perturbations. Consequently, the detected AEs are irrelevant inputs in the application context or noticeably unrealistic to humans. This may lead to a limited effect on improving the DL model’s dependability, as the testing budget is likely to be wasted on detecting AEs that are encountered very rarely in its real-life operations. </jats:p> <jats:p>In this article, we propose a new robustness testing approach for detecting AEs that considers both the feature-level distribution and the pixel-level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature-level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds is indicated by the auxiliary information, which are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm–based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.</jats:p>

Item Type:	Article
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	06 Nov 2023 17:14
Last Modified:	11 Mar 2024 03:29
DOI:	10.1145/3625290
Related URLs:	Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3176648