Temporal Consistency Constrained Transferable Adversarial Attacks with Background Mixup for Action Recognition

1 Introduction

Action recognition has established itself as a fundamental task in computer vision, and has widespread applications in many areas, such as video surveillance, robot, and virtual reality. In recent years, Deep Neural Networks (DNNs) have gained large popularity in developing action recognition models, such as SlowFast [Feichtenhofer et al., 2019] and Video Vision Transformer (ViViT) [Arnab et al., 2021], which focuses on capturing the spatiotemporal features of video. However, these model are vulnerable to adversarial examples [Goodfellow et al., 2015], i.e., adding human imperceptible perturbations to clean samples, which fool the classification model to make incorrect predictions. This raises the important security concerns from both academia and industry.

In real-world scenarios, due to some privacy issue, the prior knowledge such as the parameters and the architecture of target model (i.e., attacked model) are often unavailable to the attacker. Hence, we investigate the transferability of adversarial examples by adopting one white-box model as surrogate or source model to realize black-box attacks for action recognition models. To this end, there are only a few studies devoted to the transfer-based attack on video recognition models. For example, [Wei et al., 2022] presents a Temporal Translation (TT) attack method that optimizes the adversarial perturbations on the temporal translated video clips to make the attack less sensitive to varying temporal patterns. From the crossmodal transfer perspective of image-to-video, [Wang et al., 2023] improves the transferability of adversarial examples in black-box scenario by introducing global inter-frame interaction into attack process and disrupting the inherently local correlations of frames within a video; [Wei et al., 2024] generates adversarial examples from white-box image models to attack video models, and optimize perturbations via reducing the similarity of intermedia features between clean frame and adversarial frame. However, the above methods have two major drawbacks including: 1) they update gradients to generate adversarial examples which makes them heavily rely on the assumption that the decision boundaries of the surrogate (a.k.a., source) model and the target model are similar, i.e., the transfer-based attack will fail (e.g., adversarial example is correctly recognized by target model) when the two boundaries are isolated away (see Fig. 1), and 2) the existence of the boundary difference incurs the uncertainty problem of attack direction, and they make the attempt in many possible directions, resulting in possible large gradient oscillation as shown in Fig. 2 and thus weakening the adversarial attack.

For the first problem, from the input transformation perspective, we adopt the adversarial mixup strategy to generate adversarial examples, which makes the transferable attack less dependable on the assumption of the decision boundary similarity between the surrogate model and the target model. The so-called mixup acts as data augmentation that linearly interpolates two images and corresponding labels. Here we do not change the label as in [Wang et al., 2021a]. By contrast, we consider the semantic perturbation on clean sample by making a set of background frames, each of which corresponds to one category. In particular, we add the background frame with the highest attack reward to the clean frame. All clean frames within a video may be mixed up with different background frames. To identify the background frame with the strongest attack ability, we leverage reinforcement learning by designing a reward function for the adversarial Mixer, which includes attack success award, transfer award, and temporal background consistency award. The background frame with the top score is selected for mixing up with the clean frame. Here the temporal background consistency award makes the selected background frames be similar or the same for nearby frames. These skills are wrapped in the modelagnostic Background Adversarial Mixup (BAM) module.

For the second problem, from the temporal attack perspective, some work [Wei et al., 2022] presents a gradient-based temporal translation attack that optimizes the adversarial perturbations on temporally translated video clips, while some others [Wei et al., 2024][Chen et al., 2023] focus on featurebased attacks that minimize the cosine similarities between the intermediate features of the (warped) clean frame and the adversarial counterparts. However, they fail to consider the local relation of nearby frames. Meanwhile, [Wang et al., 2023] disrupts the temporal local correlations by reducing the similarity of the adversarial examples of nearby frames, but it introduces the large diversity of adversarial frames by minimizing their feature cosine similarity. This makes the attack directions vary greatly, leading to possible inverse gradient directions as shown in Fig. 2. Thus, it adds the difficulty in pushing the adversarial example away from the current decision boundary. Hence, we design the Background-induced Temporal Gradient enhancement (BTG) module that leverages the background attack loss and the temporal gradient consistency loss to make nearby frames have similar gradients. This makes the attack directions be consistent across the frames along the temporal dimension, enhancing the attack ability of adversarial example gradually.

We briefly summarize the main contributions as follows:

• We study the transfer-based black-box attack on action recognition models, and propose a Background Mixupinduced Temporal Consistency (BMTC) attack method to boost the transferability of adversarial examples.
• We perform a model-agnostic input transformation by considering background semantics, i.e., adversarially mixup background frame from other categories with clean frame, while that background frame is determined by the reward function using reinforcement learning with good transferability and temporal consistency.
• We strengthen the attack direction across sequential frames in a progressive way, by imposing the temporal gradient consistency constraint on the loss and guiding the attack in the direction of the background category.

2 RelatedWorks

2.1 Transfer-based Image Attacks

Previous methods often adopt white-box attacks, such as Fast Gradient Sign Method (FGSM) [Goodfellow et al., 2015] that updates the gradient on the clean sample to maximize the loss function, Projected Gradient Descent (PGD) [Madry et al., 2018] that is an iterative FGSM. To improve the transferability of adversarial examples on black-box models, there are three strategies: 1) data augmentation (i.e.,input transformation), e.g., diversity input attack [Xie et al., 2019], Scale-Invariant Method (SIM) [Lin et al., 2020], TranslationInvariant (TI) attack [Dong et al., 2019], Adversarial Mixup (AdMix) [Wang et al., 2021a] that mixes two images using Mixup [Zhang et al., 2018], and Path-Augmented Method (PAM) [Zhang et al., 2023] that constructs a candidate augmentation path pool for path selection with greedy search; 2) gradient modification, e.g., Momentum Iterative (IM) attack [Dong et al., 2018], Skip Gradient Method (SGM) [Wu et al., 2020], Variance Momentum Iterative tuning (VMI) [Wang and He, 2021], and Gradient-Related Adversarial Attack (GRA) [Zhu et al., 2023]; 3) feature disruption, e.g., Intermediate Level Attack (ILA) [Huang et al., 2019], Feature Importance-Aware attack (FIA) [Wang et al., 2021b], and ILA with Data Augmentation (ILA-DA) [Yan et al., 2023].

2.2 Transfer-based Video Attacks

Transferable attacks on video models are less explored compared to that on image models. Existing methods attempt to break the temporal relations, e.g., from the gradient update perspective, Temporal Translation (TT) [Wei et al., 2022] does multiple translations on frames along the temporal dimension to generate the clips under different translation patterns and optimizes the perturbations on these clips; from the feature disruption perspective, some methods minimize the cosine similarities between the intermediate features of the clean frames [Wei et al., 2024] or the warped clean frames

[Chen et al., 2023] and the adversarial counterparts. Nevertheless, they treat video as a collection of unordered images, ignoring the inherent temporal continuity. In contrast, [Wang et al., 2023] disrupts the temporal local correlations of nearby frames by reducing the similarity of the adversarial counterparts, and minimizes the cosine similarities among the intermediate features of the benign points lying on a convex hull, which leads to diverse adversarial frames. This may incur gradient oscillation, e.g., some sample gradients update in reverse direction, because the uncertain attack directions vary greatly.

2.3 Action RecognitionModels

Much progress has been made in deep neural network based action recognition models. Existing methods have two groups: 1) Convolutional Neural Networks (CNNs) methods, e.g., Inflated 3D ConvNet (I3D) [Carreira and Zisserman, 2017], Non-Local neural networks (NL) [Wang et al., 2018], SlowFast [Feichtenhofer et al., 2019] that captures spatial semantics at low frame rate and the motion dynamics at fine temporal resolution, Temporal Pyramid Networks (TPN) [Yang et al., 2020] that uses the source features and the fusion features to form a feature hierarchy to capture action instances at various tempos; 2) Vision Transformer (ViT) methods, e.g., VideoTransformer Network (VTN) [Neimark et al., 2021] that is built on top of any given 2D spatial network and attends to the entire video, Time-Space Transformer (TimeSformer) [Bertasius et al., 2021] that leverages 3D self-attention over the space-time volume to capture long-range temporal dependency among frames, Motionformer [Patrick et al., 2021] aggregates implicitly motion path information to model the temporal dynamic scenes, and VideoSwin [Liu et al., 2022] that encourages an inductive bias of locality by adapting the Swin Transformer [Liu et al., 2021] for a better speed-accuracy trade-off.

3 Methodology

This section introduces the proposed Background Mixupinduced Temporal Consistency (BMTC) attack method in the transfer-based black-box setting for action recognition models. The overall framework is illustrated in Fig. 3, which consists of two primary components, i.e., Background Adversarial Mixup (BAM) module and Background-induced Temporal Gradient enhancement (BTG) module.

3.1 ProblemDefinition

Given a video sample x represented by a tensor X ∈ RT×H×W×C with the ground-truth label y ∈ Y = {1, 2, · · · ,K} represented by a one-hot vector y ∈ RK , where {N,H,W,C} denote the number, height, width, and channel of the video, each frame is indexed by t ∈ {1, 2, · · · , T}, and there are K action categories, transferbased adversarial attack aims to generate an adversarial example Xadv = X+ δ by the surrogate model g(·) to fool the target model f(·) : X ↦→ Y to make incorrect predictions, i.e., f(Xadv ) ̸= y without knowing the gradients and architecture of the model, where the perturbation δ is restricted by the ℓp-norm ∥δ∥p ≤ ϵ. Here, ϵ > 0 is a constant that governs the perturbation magnitude, and we adopt ℓinf -norm and untargeted adversarial attacks as in [Wei et al., 2024][Wei et al., 2022]. The objective of untargeted adversarial attack is formulated as:

where the function J(·) often adopts cross-entropy loss. Since this work focuses on the black-box transfer-based attack, the attacker (a.k.a., adversary) has no access to details of the target model f(·). We aim to improve the black-box transferability of video adversarial examples on other action recognition models.

3.2 Background Adversarial Mixup

To ensure the good transferability of attacks, the decision boundary of the target model is expected to approach that of the surrogate model. When their decision boundaries are far away, it results in poor transferability of adversarial examples, e.g., they make different predictions on the same sample. Gradient modification methods heavily depends the above decision boundary assumption, since the gradients back-propagated during training have essential compacts on reshaping the decision boundary of the action recognition model. To circurvement this problem, we present a modelagnostic input transformation method, which adopts the adversarial mixup with background in video using a Mixer that consists of a feature extractor (e.g., ResNet50) and a classification head (e.g., fully-connected layer). The rationale behind this is that there exist strong correlations between action category and action background, e.g., surfing in the blue sea and riding on a road, which makes it possible that mixing the background from other categories with a given video might mislead the model to yield wrong predictions. This raises a key problem that how to select the background frame. The working mechanism is shown below.

We construct a set of candidate background frames, which are obtained by randomly selecting one background frame of a randomly chosen video from each category. The selected background frame is made by applying one zero-shot video object segmentation method, i.e., Isomerous transformer 1 [Yuan et al., 2023], to abandon foreground part, such as person and action-related objects. Note that the label of each background frame is kept the same as that in the original video, and there is a one-to-one correspondence of label mapping between original video and background frame, i.e., yback = y + K, whose one-hot vector is yback ∈ RK and its index is k′ = {1, 2, · · · ,K}. The background frame is repeated to form the background video Xback ∈ RT×H×W×C for a video X. These background videos are added to the set of original videos and they are fed into the surrogate model for fine-tuning, whose loss Lft is defined as

where the first term is action loss and the second term is background loss, {ˆyk, ˆyback k′ } are the probability logits from the model, the hyper-parameter λ (empirically set to 0.2) is used to balance the contribution of the two terms to the objective. Note that fine-tuning the surrogate model is to equip the surrogate model with some discriminative ability of different background categories, so the action loss should play a dominant role and the background loss serves as an auxiliary.

1 https://github.com/DLUT-yyc/Isomer

To select the background frame with the most powerful attack ability, we adopt the reinforcement learning to compute three reward functions, i.e., attack reward Rattack, transfer reward Rtransfer, and temporal background consistency reward Rtbc. These reward functions are used to encourage the Mixer to generate the admixed sample to successfully attack the surrogate model. Here the Mixer is used to mixup the selected background video with the original video, i.e., ˜X = (1 - γ) · X+ γ · Xback , where the constant γ is empirically set to 0.2. For the frame-level mixup, the i-th admixed frame is ˜Xi = (1 - γ) · Xi + γ · Xback i . Actually, admixed sample serves as the vanilla adversarial example.

Attack Reward. For a video x with its label y and the predicted probability vector ˆy = [ˆ y1, ˆ y2, · · · , ˆ yK], this reward Rattack takes the difference of the label prediction probabilities before and after the adversarial mixup, i.e.,

where {ˆ y\k} is a set excluding the k-th entry, which indicates the true label k. When the probability of the true label ˆ yk increases, the reward score decreases; when the maximum probability of other categories max{ˆ y\k} increases, the reward score rises, i.e., when the video is incorrectly predicted, we should give the model positive reward.

Transfer Reward. To improve the transferability of adversarial example across different models, we provide the transfer reward Rtransfer to attack M black-box target models, and averages their attack rewards, i.e.,

where Rm attack is the attack award of the m-th target model.

Temporal Background Consistency Reward. Since similar samples have similar backgrounds, nearby frames are expected to have similar backgrounds. For a video x with T frames indexed by t, the neighbouring frames are expected to be mixed with similar or the same background frames, i.e., temporal consistency among background frames, such that the attack strength will be boosted. We define the Temporal Background Consistency (TBC) reward Rtbc as

where ∥ · ∥2 denotes the ℓ2-norm, and the vector ˆyback t ∈ RK is the predicted probability of the t-th background frame.

Therefore, the total reward of selecting the background frame is

where the constants α1 > 0 (set to 0.3) and α2 > 0 (set to 0.1) are used to control the contributions of the transfer reward and the temporal background consistency reward, respectively. Finally, we select the background frame with the highest total reward score from K candidates to mixup with the original video frames sequentially.

3.3 Background-induced Temporal Gradient Enhancement

To further improve the transferability of vanilla adversarial example (i.e., admixed sample), we introduces two adversarial losses including the background attack loss and the temporal gradient consistency loss. The former is used to guide the surrogate model to generate the adversarial example, by learning the probability logits along the direction of the original action category, which is associated with the mixed background frame. In particular, we maximize the cross-entropy loss of other categories and minimize that of the background category (i.e., fool the model to wrongly predict the sample as the selected background category), i.e.,

where the background frame belongs to the (k′ -K)-th category from the original K actions.

To makes the attack direction be consistent between nearby adversarial frames, we design the Temporal Gradient Consistency (TGC) loss to encourage the surrogate model to generate the adversarial counterparts with similar gradients for nearby frames. By this means, the attack strength is gradually enhanced across the frames along the temporal dimension. Mathematically, we adopt the cosine similarity to evaluate the temporal consistency between the gradients of two nearby adversarial frames, i.e.,

where gt ∈ RH·W·3 denotes the gradient vector of the t-th adversarial frame Xadv t ∈ RH×W×C , which is calculated as

where θ denotes the model parameter set, vec(·) vectorizes a tensor to the corresponding vector, and ∇Xadv t L is the gradient of the loss Lback w.r.t.the t-th frame.

Therefore, the total temporal adversarial attack loss is:

where the hyper-parameter β ∈ [0, 1] (set to 0.1) controls the balance between the two terms during optimization.

3.4 Adversarial Example Generation

This work adopts the Projected Gradient Descent (PGD) algorithm [Madry et al., 2018] to compute the sample gradients of the model, and the adversarial example at the i-th iteration is updated in the following way:

where η > 0 is the step size, ΠX,ϵ[·] = min(max(Xi ,Xϵ),X + ϵ) is the projection function that controls the perturbation magnitude, X is the clean sample, and the initial X0 is the admixed sample. When the iteration achieves the maximum number Niter (e.g., set to 10 for a good tradeoff between performance and efficiency), we obtain the ultimate adversarial example to attack the target models.

4 Experiments

All experiments were performed on a server equipped with two 48G NVIDIA A6000 graphics cards. The codes are compiled with PyTorch 1.11, Python 3.8, and CUDA 11.4. Our code is available at Github.

4.1 Datasets and EvaluationMetric

We conduct experiments on two video benchmarks including UCF101 [Soomro et al., 2012] and Kinetics-400 [Carreira and Zisserman, 2017], and one image benchmark ImageNet [Deng et al., 2009]. Following [Wei et al., 2022][Wei et al., 2024], we adopt the Attack Success Rate (ASR) as the metric, which calculates the rate of adversarial examples misclassified by the black-box model. The higher the ASR, the better the transferability is. As a common practice, one video is randomly selected from each category and correctly classified by all target models.

4.2 Experimental Settings

For the adversarial Mixer, we train a classification model from scratch with ResNet50 as backbone on Kinetics-400 and ImageNet respectively, and the training epochs are 100. For the surrogate model fine-tuning, we adopt TPN and ResNet101 as the surrogate of video (20 epochs) and image (10 epochs) attack, respectively, and the hyper-parameter of the fine-tuning loss is λ = 0.2. For both of them, the initial learning rate is 0.1, momentum is 0.9, and the weight decay is 1e4. For the adversarial example generation, we set the maximum perturbation ϵ to 16 for video and 8 for image, the attack step size η to 1.6 for video and 0.8 for image, the maximum iteration number Niter to 10.

For action recognition, we examine three CNN models with different architectures, i.e., Non-local network (NL) [Wang et al., 2018], SlowFast (SF) [Feichtenhofer et al., 2019], and Temporal Pyramid Networks (TPN) [Yang et al., 2020], which employ ResNet50 and ResNet101 as backbones. We train video models from scratch with Kinetics-400 and fine-tune them on UCF101. Following [Wei et al., 2024], we skip every other frame from randomly selected clip with 64 consecutive frames to make input for Kinetics-400, and use 32 consecutive frames as input clips for UCF101. Moreover, we also examine four Transformer models including VideoTransformer Network (VTN) [Neimark et al., 2021], Time-Space Transformer (TimeSformer) [Bertasius et al., 2021], Motionformer [Patrick et al., 2021], and VideoSwin [Liu et al., 2022]. Following [Wei et al., 2024], we sample one clip per video and each clip consists of 16 frames with the temporal stride 4. For image classification, we examine four models including ResNet18 [He et al., 2016], ResNet101, ResNeXt50 (RNX50) [Xie et al., 2017], and DenseNet121 (DN121) [Huang et al., 2017]. Note that NL-101, SF-101 and TPN-101 using ResNet101 act as white-box models (surrogate) for generating adversarial examples to attack other black-box models (target). The input spatial size is 224×224. The model number M is 3 for video and 4 for image.

4.3 Quantitative Results

Action Recognition. The attack results of action recognition models are reported in Table 1 on UCF101, Tables 2 and 3 on Kinetics-400. The best records are highlighted in boldface, and the second best ones are underlined. Here, "RN" denotes ResNet, the records with the gray color are the results of white-box attack to be overlooked.

From these tables, we observe that our method achieves more satisfying performance across several action recognition models including NL, SlowFast, and TPN, with different architectures. For example, when using the adversarial example from SF-101 to attack NL and TPN with RN50 as backbone, our method has a gain of 9.0% and 21.0% respectively, in comparison to TT on Kinetics-400; when using the adversarial example from TPN-101 to attack NL and SlowFast with RN50 as backbone, our method has an improvement of 13.87% and 22.78% respectively, compared to the most competitive alternative TT on UCF101. This demonstrates the superiority of the adversarial mixup strategy with reinforcement learning and the temporal consistency among nearby frames. Besides CNN models, the performance improvements are also found on ViT models in Table 3.

Meanwhile, the cross-modal attack results in terms of ASR are shown in Table 4 on UCF101 and Table 5 on Kinetics400. Here, the surrogate models of the compared methods are AlexNet [Krizhevsky et al., 2012], ResNet-101 [He et al., 2016], SqueezeNet [Iandola et al., 2016], and VGG-16 [Simonyan and Zisserman, 2015] in image domain, while we use the other two video models as the surrogate models, e.g., SF and TPN as surrogate and NL as target (group 1). The results are averaged over those surrogate models, which indicates the advantages of the proposed attack method.

Image Classification. The results of image classification models are reported in Table 6 on ImageNet. Here, "Ori" denotes the vanilla attack method (column 2). Note that the temporal consistency reward and background-induced loss fail in image domain. Even though only using the adverse award and the transfer award to select the background, ours still improves the attack performance of the vanilla ones. This once again validates the power of the adversarial mixup with the background frame.

4.4 Ablation Study

We report the average ASR over black-box attacks and the hyper-parameters keep still as in training unless specified.

Individual component. The results are shown in Table 7, where the baseline is vanilla PGD, which is very poor as it neglects temporal cues. When adding the background attack loss (row 2), the performance is doubled; when using the Background Adversarial Mixup (BAM) module (row 3) or Background-induced Temporal Gradient enhancement (BTG) module (row 4), the performance is largely improved and the former is better. When using both of BAM and BTG (Ours), the performance achieves the best. When abandoning the reward Rtbc (row 5), the performance degrades, which demonstrates the background consistency along the temporal dimension affects the total reward that decides the selected background frame. When abandoning Ltgc (row 6), the attack performances deteriorate significantly by about 7% to 12%, which validates the importance of gradient consistency between two nearby adversarial frames.

Niter and ϵ. The results are shown in Table 8, which shows that the attack performance is naturally improved with the increasing iteration number Niter and the maximum perturbation ϵ. However, we observe that when Niter rises from 10 to 20, the performance becomes stable and even suffers from the bottleneck (row 2/4 in group 1) at much larger computational cost. So we choose 10 for Niter. Besides, while the performance is greatly boosted by using larger perturbations, e.g., ϵ=32, the adversarial examples are easily found by human, which violates the attack rule that requires small human-imperceptible noise. So we use the trade-off 16 for ϵ.

γ and β. The results are shown in Table 9, where the attack performance rises up when γ in the adversarial mixup increases from 0.1 to 0.2 and then decreases when γ is over 0.2. This demonstrates that the background frame should not dominate the mixup. Similar observations are found for the hyper-parameter β of the temporal gradient consistency loss Ltgc, when it starts from 0.01 to 0.1. This suggests that neither too small nor large values are taken for β.

4.5 Computational Efficiency

We report the parameter size, the computational cost (GFLOPs), the inference speed (fps), and the ASR score on UCF101 and Kinetics-400 in Table 10. From the table, our BMTC method enjoys a more satisfying tradeoff between performance and efficiency. For example, compared to Temporal Translation (TT) [Wei et al., 2022], ours have much higher ASR score (69.31 vs 52.47 on UCF101, 75.69 vs 62.75 on Kinetics-400) with only nearly one-tenth GFLOPs of TT

at a 16 times faster inference speed.

4.6 Visualization of Adversarial Examples

To visualize the performance of our attack, we randomly chose one video from UCF101 and Kinetics-400, respectively, and show their adversarial examples in Fig. 4 (zoom in for better view). Compared to the baseline Temporal Translation (TT) [Wei et al., 2022], the perturbations on the adversarial examples of ours are almost imperceivable by human but the model can be fooled to make wrong predictions.

5 Conclusion

This work presents a transferable black-box adversarial attack method for action recognition by considering both the temporal background consistency and the temporal gradient consistency. In particular, we adopt the adversarial mixup strategy to mix the clean sample with the background frame from other categories. To ensure the attack ability of the background frame, we design a reward function that considers the temporal consistency among nearby frames and the transferability across different models. Moreover, we strengthen the transferability of adversarial example by adopting the background-induced temporal consistency on the gradients of sample across frames. Empirical studies on both video and image datasets validate the effectiveness of the proposed attack on several models with different architectures.

Acknowledgments

This work was supported in part by Zhejiang Provincial Natural Science Foundation of China under Grant LR23F020002, and in part by Hangzhou Key Research and Development Program under Grant 2024SZD1A12.