Yi, Xinping 
ORCID: 0000-0001-5163-2364
(2022)
Asymptotic Spectral Representation of Linear Convolutional Layers.
IEEE Transactions on Signal Processing, 70.
pp. 566-581.
|
Text
final.pdf - Author Accepted Manuscript Download (564kB) | Preview |
Abstract
By stacking a number of convolutional layers, convolutional neural networks (CNNs) have made remarkable performance boosts in many artificial intelligence applications. While the convolution operation is well-understood, it is still a mystery why repeated convolutions yield so good expressive power and generalization performance. Noting that the linear convolution operation can be represented as a matrix-vector product with the matrix being of a Toeplitz structure, we propose to inspect the individual convolutional layer through its asymptotic spectral representation - the spectral density matrix - by leveraging Toeplitz matrix theory. Thanks to such spectral representation, we are able to develop a simple singular value approximation method with improved accuracy, and spectral norm upper bounds with reduced computational complexity, compared with the state-of-the-art methods. Both the improved approximation and upper bounds can be employed as regularization techniques to further enhance the generalization performance of CNNs. By extensive experiments on well-deployed CNN models (e.g., ResNets), we also demonstrate that the approximation approach achieves higher accuracy and the upper bounds are effective spectral regularizers for generalization.
| Item Type: | Article |
|---|---|
| Divisions: | Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 12 Jan 2022 15:58 |
| Last Modified: | 17 Mar 2024 13:17 |
| DOI: | 10.1109/tsp.2022.3140718 |
| Related URLs: | |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3146128 |
Dimensions
Dimensions
