Structural Test Coverage Criteria for Deep Neural Networks



Sun, Youcheng, Huang, Xiaowei ORCID: 0000-0001-6267-0366, Kroening, Daniel, Sharp, James, Hill, Matthew and Ashmore, Rob
(2019) Structural Test Coverage Criteria for Deep Neural Networks. ACM Transactions on Embedded Computing Systems, 18 (5S). pp. 1-23.

[img] Text
emsoft2019.pdf - Author Accepted Manuscript

Download (1MB) | Preview

Abstract

Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that test inputs that are generated with guidance by our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test input generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet.

Item Type: Article
Uncontrolled Keywords: Neural networks, test criteria, test case generation
Depositing User: Symplectic Admin
Date Deposited: 13 Jan 2020 15:41
Last Modified: 19 Jan 2023 00:10
DOI: 10.1145/3358233
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3070444