Enhancing Adversarial Training with Second-Order Statistics of Weights

Jin, Gaojie, Yi, Xinping ORCID: 0000-0001-5163-2364, Huang, Wei, Schewe, Sven ORCID: 0000-0002-9093-9518 and Huang, Xiaowei ORCID: 0000-0001-6267-0366 (2022) Enhancing Adversarial Training with Second-Order Statistics of Weights. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022-6-18 - 2022-6-24.

Text
CVPR_2022_Second_Order_Statistics_of_Weights.pdf - Author Accepted Manuscript
Download (8MB) | Preview

Official URL: http://dx.doi.org/10.1109/cvpr52688.2022.01484

Abstract

Adversarial training has been shown to be one of the most effective approaches to improve the robustness of deep neural networks. It is formalized as a min-max optimization over model weights and adversarial perturbations, where the weights can be optimized through gradient descent methods like SGD. In this paper, we show that treating model weights as random variables allows for enhancing adversarial training through Second-Order Statistics Optimization (S2O) with respect to the weights. By relaxing a common (but unrealistic) assumption of previous PAC-Bayesian frameworks that all weights are statistically independent, we derive an improved PAC-Bayesian adversarial generalization bound, which suggests that optimizing second-order statistics of weights can effectively tighten the bound. In addition to this theoretical insight, we conduct an extensive set of experiments, which show that S2O not only improves the robustness and generalization of the trained neural networks when used in isolation, but also integrates easily in state-of-the-art adversarial training techniques like TRADES, AWP, MART, and AVMixup, leading to a measurable improvement of these techniques. The code is available at https://github.com/Alexkael/S2O.

Item Type:	Conference or Workshop Item (Unspecified)
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	05 Apr 2022 08:28
Last Modified:	18 Jan 2023 21:05
DOI:	10.1109/CVPR52688.2022.01484
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3152097