A Quantitative Insight Into the Role of Skip Connections in Deep Neural Networks of Low Complexity: A Case Study Directed at Fluid Flow Modeling

Choubineh, Abouzar, Chen, Jie, Coenen, Frans ORCID: 0000-0003-1026-6649 and Ma, Fei ORCID: 0000-0001-6099-480X (2023) A Quantitative Insight Into the Role of Skip Connections in Deep Neural Networks of Low Complexity: A Case Study Directed at Fluid Flow Modeling. JOURNAL OF COMPUTING AND INFORMATION SCIENCE IN ENGINEERING, 23 (1).

Text
abouzarJCISE-2022.pdf - Author Accepted Manuscript
Download (1MB) | Preview

Official URL: http://dx.doi.org/10.1115/1.4054868

Abstract

<jats:title>Abstract</jats:title> <jats:p>Deep feed-forward networks, with high complexity, backpropagate the gradient of the loss function from final layers to earlier layers. As a consequence, the “gradient” may descend rapidly toward zero. This is known as the vanishing gradient phenomenon that prevents earlier layers from benefiting from further training. One of the most efficient techniques to solve this problem is using skip connection (shortcut) schemes that enable the gradient to be directly backpropagated to earlier layers. This paper investigates whether skip connections significantly affect the performance of deep neural networks of low complexity or whether their inclusion has little or no effect. The analysis was conducted using four Convolutional Neural Networks (CNNs) to predict four different multiscale basis functions for the mixed Generalized Multiscale Finite Element Method (GMsFEM). These models were applied to 249,375 samples. Three skip connection schemes were added to the base structure: Scheme 1 from the first convolutional block to the last, Scheme 2 from the middle to the last block, and Scheme 3 from the middle to the last and the second-to-last blocks. The results demonstrate that the third scheme is most effective, as it increases the coefficient of determination (R2) value by 0.0224–0.044 and decreases the Mean Squared Error (MSE) value by 0.0027–0.0058 compared to the base structure. Hence, it is concluded that enriching the last convolutional blocks with the information hidden in neighboring blocks is more effective than enriching using earlier convolutional blocks near the input layer.</jats:p>

Item Type:	Article
Uncontrolled Keywords:	deep neural network, backpropagation, vanishing gradient phenomenon, skip connection, GMsFEM, heterogeneous porous media
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	19 Jul 2022 14:06
Last Modified:	24 Jan 2023 08:27
DOI:	10.1115/1.4054868
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3158295