Coverage Guided Testing for Recurrent Neural Networks



Huang, wei, Sun, Youcheng, Zhao, Xingyu ORCID: 0000-0002-3474-349X, Sharp, James, Ruan, Wenjie, Meng, Jie and Huang, Xiaowei ORCID: 0000-0001-6267-0366
(2021) Coverage Guided Testing for Recurrent Neural Networks. IEEE Transactions on Reliability.

WarningThere is a more recent version of this item available.
[img] Text
testRNN_accepted_version.pdf - Author Accepted Manuscript

Download (3MB) | Preview

Abstract

Recurrent neural networks (RNNs) have been applied to a broad range of applications including natural language processing, drug discovery, and video recognition. However, their vulnerability to input perturbation is also exposed. Aligning with a view from software defect detection, this paper aims to develop a coverage guided testing approach to systematically exploit the internal behaviour of RNNs, with high possibility of detecting defects. Technically, the long short term memory network (LSTM), a major class of RNN, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both step-wise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. Based on these, we develop a tool TESTRNN, and extensively evaluate TESTRNN on a set of LSTM benchmarks. Experiments confirm that TESTRNN has several advantages over the state-of-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, TESTRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step towards interpretable neural network testing.

Item Type: Article
Divisions: Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User: Symplectic Admin
Date Deposited: 10 May 2021 10:16
Last Modified: 18 Jan 2023 22:49
URI: https://livrepository.liverpool.ac.uk/id/eprint/3121687

Available Versions of this Item