Enhancing Transferability of Audio Adversarial Example for Both Frequency- and Time-domain

1 Introduction

Deep neural networks (DNNs) have achieved exceptional performance in audio classification tasks [Sadovsky et al., 2023; Liu et al., 2024; Choi et al., 2025; Gazneli et al., 2022; Berg et al., 2021]. However, they have been demonstrated to be vulnerable to audio adversarial examples, which are maliciously crafted by imposing acoustically imperceptible perturbations on clean audio examples to result in erroneous predictions [Goodfellow et al., 2015; Hai et al., 2023; Huang et al., 2023; Ma et al., 2023a; Jin et al., 2024]. Numerous works suggest adversarial examples exhibit cross-model transferability, i.e., those crafted against one surrogate model can also mislead other models. This property makes blackbox attacks practically effective, posing significant threats to safety-critical applications. This paper focuses on investigating the transferability of audio adversarial examples because of their practicality.

∗ Corresponding author.

There are two approaches, gradient-optimization [Fang et al., 2024; Zhu et al., 2023; Zhang et al., 2023] and model ensemble [Tang et al., 2024; Chen et al., 2023a; Ma et al., 2023b] to improve the transferability of adversarial examples in the image field. Transferable audio adversarial attacks are inspired by these similar methods [Tripathi and Mishra, 2022; Abdoli et al., 2019; Koerich et al., 2020]. However, we observe that the generated audio adversarial examples face a critical limitation, i.e., they exhibit low transferability across different domains, specifically the time and frequency domains [Chen et al., 2024]. Typically, audio classification models are categorized into time-domain models [Wang et al., 2023; Zeng et al., 2021; Abdallah et al., 2022] and frequency-domain models [Gong et al., 2021; Liu et al., 2024; He et al., 2024] according to the input form. We find that audio adversarial examples generated by attacking a time domain surrogate model exhibit very limited transferability to frequency domain black-box models, as illustrated in Figure 1a. This limitation could be attributed to changes in data distribution introduced by the time-frequency transformation and the architectural differences between models. Moreover, the nonlinearity of the time-frequency transformation makes the exact inversion of perturbations from spectrograms back to audio intractable, consequently, adversarial examples crafted directly in the frequency domain are less likely to transfer to time-domain models, as shown in Figure 1b.

The above observations motivate us to dive into the question: How can collective transferable adversarial information from both domains be effectively captured and directly used to generate perturbations on the audio waveform? To cope with this problem, we investigate performing an interdomain ensemble strategy, which fuses outputs from both domains to get an ensemble loss. We demonstrate that the time-frequency transformation, such as transforming a Melspectrogram back to a waveform, is differentiable, enabling the gradient of the ensemble loss to be propagated back to the audio waveform. Therefore, the generated adversarial examples would be optimized to converge toward a common adversarial space among both domains. However, we observe that simply averaging the outputs of both domains causes ad-

(a) Only time-domain model

(c) Inter-domain Ensemble

(b) Only frequency-domain model

(d) Adaptive Inter-domain Ensemble

Figure 1: The attack success rates with different surrogate models. (a) and (b) are the MI-FGSM using a surrogate model from the time and frequency domains, respectively. (c) and (d) are the Interdomain Ensemble (IE) and Adaptive Inter-domain Ensemble (AIE), which use the time and frequency surrogate models. The dataset is UrbanSound8k.

versarial examples to predominantly transfer to one domain, while their transferability to the other domain decreases compared to ones crafted against the uni-domain case, as depicted in Figure 1c. Such a phenomenon violates the intention of improving transferability by integrating information from both domains. Analyzing the evolution tendency of adversarial examples during the update process, we argue that this update progression is often dominated by one domain, suppressing the transferability to the other one. Hence, we further propose an adaptive domain weight adjustment method. Specifically, the weight assigned to each domain's output is dynamically adjusted by monitoring the ratio of the cosine similarity between the potential outputs of each domain and the ensemble potential output. We term the proposed inter-domain ensemble attack with adaptive domain weight adjustment as the Adaptive Inter-domain Ensemble (AIE) attack, as illustrated in Figure 2. It mitigates the above imbalance and enhances cross-domain transferability, as shown in Figure 1d. Our contributions can be summarized as follows:

• To the best of our knowledge, this is the first work aimed at enhancing the cross-domain transferability of audio adversarial examples to simultaneously deceive classifiers in both the time and frequency domains.
• We propose an Adaptive Inter-domain Ensemble attack that captures collective transferable adversarial information from both domains while dynamically assigning the weight to each domain, enhancing the cross-domain transferability of audio adversarial examples.
• Extensive experiments on diverse datasets demonstrate that AIE consistently outperforms existing methods,

the generated audio adversarial examples exhibit higher cross-domain transferability. Notably, AIE can be integrated with various gradient optimization attacks.

2 Overview

2.1 Preliminaries

Given a clean audio x with its corresponding ground-truth label y, the audio classification model f is expected to predict label argmaxf(x) = y with high confidence, where f(x) represents logit output. Let Bϵ (x) = {ˆx : ∥ˆx-x∥p ≤ ϵ} be an ϵ-ball around x, where ϵ > 0 is a predefined perturbation magnitude, and ∥ · ∥p denotes the Lp-norm (e.g., L1-norm). Transfer-based adversarial attacks aim to find an example xadv ∈ Bϵ that misleads the model prediction argmaxf(xadv ) ̸= y, and then transfer xadv to directly attack the black-box target model. The objective can be formulated as the following constrained optimization problem:

where L(·) is the cross-entropy function. The typical process of generating adversarial examples through gradient iterations can be described as:

where xadv 0 = x, t is the current number of iteration, α is the step size, and Φϵ projects xadv t into the ϵ-ball around x.

2.2 Motivation

We observe that existing methods for crafting transferable audio adversarial examples face a critical limitation, i.e., they overlook the transferability across different domains, specifically the time and frequency domains. Typically, audio classification models are categorized into the following two types based on input form: time-domain model, denoted as fw, which uses raw waveforms of audio x as input, and frequency-domain model, denoted fs, which utilizes spectrograms F(x) obtained through the time-frequency transformation F(·). Normally, for a clean audio x, the predictions of the two models are expected to be consistent, which can be expressed as:

Both time-domain and frequency-domain models are vulnerable to adversarial attacks. However, due to differences in data distribution and model structures, existing attacks demonstrate low cross-domain transferability. We notice that the adversarial example xadv generated by attacking the timedomain model fw exhibits very limited transferability to the frequency-domain model fs after being converted into spectrograms F(xadv ). As the exact inversion of perturbations on spectrograms is intractable due to the nonlinearity of the timefrequency transformation, crafting audio adversarial examples by attacking frequency-domain models remains a largely unsolved challenge. From a practical standpoint, the initial form of input to all audio classification models is raw waveforms. Therefore, a true black-box attack on the audio classification task must directly generate perturbations on waveforms and be able to fool target models across different domains. The final attack objective can be described as:

3 Adaptive Inter-domain Ensemble Attack 3.1 Uniformly Inter-domain Ensemble

To make generated adversarial examples more likely to transfer to models from both time and frequency domains, we propose an Inter-domain Ensemble attack, which fuses outputs from the time and frequency domains to get an ensemble loss, finding a common adversarial space among them. We begin by simply ensembling two models fw and fs. For the xadv , the fusion output can be represented as

where η ∈ [0, 1] is the domain weight, initially set to 0.5. Intuitively, only the gradient of fw can be backpropagated to xadv . However, we introduce a key property of the timefrequency transformation: differentiability.

Proposition 1. Differentiability of time-frequency transformation. Given an audio x with its corresponding groundtruth label y, the frequency-domain model fs takes the spectrogram F(x) as input. The gradient propagation of the loss L(fs(F(x)), y) with respect to x can be expressed as a chain as follows:

The gradients of the time-domain and frequency-domain models in the ensemble loss can be reliably propagated backward. Consequently, the iterative process of inter-domain ensemble attack can be formulated as:

The proposed attack paradigm can leverage collective vulnerabilities of the time-domain model and frequency-domain model and generate perturbation on raw waveform, effectively boosting the cross-domain transferability of audio adversarial examples. However, we observe that simply averaging the outputs of the time domain and frequency domain, i.e., η = 0.5, causes the transferable information to be dominated by one domain, ultimately leading to suboptimal results. Let us take Figure 1 as an example again, the attack success rate decreases by 26.0% when uniformly attacking fw and fs, compared to attacking only fs (η = 0). To mitigate this problem, we further propose an adaptive adjustment strategy.

3.2 Attack Imbalance Analysis

Audio data typically exhibits high dimensionality (e.g., 22,050 Hz). Due to the curse of dimensionality, gradient similarity is unsuitable as a metric to observe the discrepancy in adapting to different domains during the updating process of the inter-domain ensemble attack, e.g., their cosine similarity tends to approach zero. Therefore, we consider calculating the similarity between outputs for evaluating the discrepancy. Specifically, for models fw and fs, we define the individual potential outputs using the probability outputs on adversarial examples generated through one-step attacks, expressed as,

We then define the ensemble potential output as the average probability output on adversarial examples generated using the gradient of the inter-domain ensemble loss with respect to the current adversarial examples xadv t , expressed as,

where η is initially set to 0.5, indicating that the time domain and frequency domain are given equal weight.

The discrepancy between the individual potential outputs and the ensemble potential output reflects the extent of dom- Figure 3: (a) shows the probability densities of cs and cw, which is imbalance. (b) illustrates the influence of the parameter of k on the adjustment function.

inance of models in the attack process. We evaluate the discrepancy using the cosine similarity,

We statistically compared the probability densities of cw and cs in the normal attack process, as shown in Figure 3a. Obviously, the updating progress of the adversarial example is often dominated by one domain (e.g., time domain).

3.3 Adaptive DomainWeight Adjustment

To address the above optimization imbalance problem, we propose to adaptively adjust the weights of each domain η via monitoring the ratio between cw and cs. Here, we define the ratio as ρ = cs/cw. With ρ to dynamically monitor the dominant discrepancy between time and frequency domains, we are able to adaptively modulate the domain weight η via:

where k is a hyperparameter that controls the slope of the function. Figure 3b intuitively illustrates how the value of η varies with the ρ under the different values of k. Constantly, η is constrained to the interval [0, 1] following its intrinsic properties. In particular, when ρ = 1, indicating an equal contribution from both domains, η converges to 0.5.

By employing the domain weight adjustment strategy, the attack process against both domain models can be effectively regulated, thereby mitigating the imbalance issue and enhancing transferability to time and frequency domains. Our AIE method can integrate existing transfer-based gradient attacks to enhance cross-domain attack performance. For example, AIE incorporating MI-FGSM [Liu et al., 2016] is summarized in Algorithm 1.

3.4 Intra-domain Ensemble Enhancement

Following Algorithm 1, AIE improves the cross-domain transferability of audio adversarial examples. While utilizing a single model from each domain demonstrates sufficient performance, we further propose augmenting AIE with an intra-domain ensemble. This approach averages the outputs of multiple models within the same domain, capturing additional intrinsic transferable adversarial information. It transforms Eq. 5 into the following representation:

Algorithm 1 AIE with MI-FGSM attack

Input: Surrogate models fw, fs; A natural audio example x with label y

Parameter: The perturbation magnitude ϵ; the number of iteration T ; the decay factor µ; the hyper-parameter k

Output: An adversarial example xadv

• 1: Initialize: α = ϵ/T;M0 = 0; xadv 0 = x;
• 2: for t = 1 to T do
• 3: Initialize set η = 0.5
• 4: #Calculate discrepancy ratio between the individual potential outputs and the ensemble potential output
• 5: Calculate the individual potential outputs ps and pw using Eq. 8
• 6: Calculate the ensemble potential output p using Eq. 9
• 7: Calculate the discrepancy ratio ρ = cos(ps,p) cos(pw,p)
• 8: #Adaptively adjust the domain weights based on the discrepancy ratio
• 9: Update the domain weight η using Eq. 10
• 10: Calculate the inter-domain ensemble loss L (E(xadv , η), y) with updated η
• 11: #Update momentum using the gradient of interdomain ensemble loss
• 12: GetMt+1 = µMt + ∇xadv t L(xadv t ,η) ∥∇xadv t L(xadv t ,η)∥1
• 13: #Update adversarial example
• 14: xadv t+1 = ∏Bϵ(x) [xadv t +α · sign (Mt+1)]
• 15: end for
• 16: return xadv = xadv T .

where f iw and f i s represent i-th surrogate model from time domain and frequency domain, respectively, and E(·) represents the average of gradients.

Moreover, such an ensemble can be seamlessly integrated with AIE without conflict, as the weight assigned to the interdomain ensemble output can be adaptively optimized. While extending AIE to incorporate a multi-model ensemble within each domain is straightforward, this extension further enhances overall transferability compared to single-model AIE, particularly in scenarios involving cross-architectural transfer. Empirical results in Section 4.3 validate the effectiveness of this ensemble-based approach.

4 Experiments

4.1 Experimental Setup

Dataset. To comprehensively evaluate the effectiveness of the proposed method, we conduct extensive experiments on two widely recognized datasets for audio classification tasks: UrbanSound8k [Salamon et al., 2014] for environmental sound classification and ShipsEar [Santos-Dom´ınguez et al., 2016] for underwater acoustic target identification.

Pre-processing. All audio examples are standardized to a length of 1 second. From each dataset, we randomly select 1000 clean audio examples, ensuring that each is correctly classified by all evaluated models and preventing data overlap. The results of frequency-domain attacks are reported us- Res-50FShufF

Res-50FShufF

Res-50FShufF

Res-50FShufF

Res-50FShufF

Res-50FShufF

Res-50FShufF

Res-50FShufF

Figure 4: The attack success rates (%) against time and frequency domain models, exploring the transferability of adversarial examples generated by AIE with various baseline methods. The first row presents UrbanSound8k results, while the second row is on the ShipsEar.

ing the Mel-spectrogram, which is generated by applying the short-time Fourier transform to the audio and then mapping the output using the Mel scale.

Baselines. Our approach is integrated and evaluated against four advanced gradient-based adversarial attacks: MI [Liu et al., 2016], NI [Lin et al., ], VMI [Wang and He, 2021], and EMI [Wang et al., 2021]. Furthermore, we compare our method with state-of-the-art ensemble attacks, namely SVRE [Xiong et al., 2022], AdaEA [Chen et al., 2023a], and CWA [Chen et al., 2023b], which have demonstrated strong transferability in the image field. These baseline methods are meticulously fine-tuned to ensure optimal performance in generating audio adversarial examples.

Models. We evaluate our approach on a diverse range of models in the time domain and frequency domain. These include Convolutional Neural Network (CNN)-based models such as Res18T and Res18F (ResNet18) [He et al., 2016], Res50T and Res50F (ResNet50) [He et al., 2016], DensT and DensF (DenseNet121) [Huang et al., 2017], EffiT (EfficientNet) [Tan and Le, 2019], and ShufF (ShuffleNetV2) [Ma et al., 2018]. Additionally, we consider a Transformerbased model ASTF (Audio Spectrogram Transformer) [Gong et al., 2021], and a Recurrent Neural Network-based model, LSTMT [Sang et al., 2018]. The subscripts T and F denote the models from the time domain and frequency domain, respectively. Furthermore, to assess robustness under adversarial conditions, we include adversarially trained models, Resadv T and Resadv F [Goodfellow et al., 2015], which are based on the ResNet18 architecture.

Hyper-parameters. We empirically set the maximum perturbation to 0.01 (l∞ = 0.01), the number of iterations T = 10, the step size α = 0.002. For MI and NI, we set the decay factor µ = 1.0. For VMI, we set the number of sampled examples N = 20 and the upper bound of neighborhood size

β = 1.5 × ϵ. For EMI, we set the number of sampled examples to 11, the sampling interval bound to 7, and adopt the linear sampling. The inner update time in SVRE is set to be four times the number of models. The tolerance threshold and temperature coefficient in AdaEa are set to be -0.3 and 10. Metrics for evaluation. We adopt the attack success rate as the unified metric to evaluate the performance of various adversarial attack methods, where 'Avg.' denotes the average attack success rate across all target models.

4.2 Attack Results

We evaluate the attack performance of audio adversarial examples generated by integrating our proposed method with the MI, NI, VMI, and EMI methods under surrogate models from both time and frequency domains. Table 1 provides a comprehensive comparison of results across multiple victim models, using Res18T and Res18F-both CNN-based architectures-as surrogate models. We observe that attacks utilizing uniformly Inter-domain Ensemble (IE) achieve significantly lower average success rates in the frequency domain compared to the time domain, highlighting the inherent limitations of achieving unbalanced transferability. In contrast, our Adaptive Inter-domain Ensemble (AIE) method consistently achieves the highest attack success rates across both domains. Specifically, when VMI is employed as the baseline attack method, AIE enhances the average attack success rate by 7.1% in the frequency domain, with an even more substantial improvement of 17.8% when attacking Res18F . These findings highlight the exceptional inter-domain transferability of adversarial examples generated by AIE and demonstrate its effectiveness in reducing the gap between transferability to time and frequency domains.

Furthermore, we evaluate the attack performance of adversarial examples generated using surrogate models with dissimilar architectures-Res18T (CNN-based) for the time domain and ASTF (Transformer-based) for the frequency domain, as depicted in Figure 4. The first row of the figure corresponds to results on the UrbanSound8k dataset, while the second row represents the ShipsEar dataset. Figure 4 consistently demonstrates that our AIE method significantly achieves higher attack success rates in both the time and frequency domains. These results highlight the ability of AIE to effectively balance and enhance transferability to both domains, regardless of differences in the model architectures.

Additionally, Figure 5 visualizes the waveforms and Melspectrograms of audio adversarial examples generated by our proposed method and MI, NI, VMI, and EMI methods under the IE strategy. The first and third rows depict the waveforms, while the second and fourth rows present the corresponding Mel-spectrograms. Although adversarial examples generated by baseline methods are similar to the clean input in both time and frequency domains, they exhibit limited effectiveness in deceiving the black-box models in the frequency domain. In contrast, our method produces adversarial examples that are perceptually similar to those of baseline attacks in both domains but achieve higher cross-domain transferability.

4.3 Evaluate Intra-domain Ensemble

AIE effectively enhances cross-domain transferability by leveraging a single surrogate model for each domain. Previous studies [Xiong et al., 2022; Chen et al., 2023a; Chen et al., 2023b] have demonstrated that incorporating more surrogate models is a powerful strategy for further improving attack transferability. To illustrate the scalability of our approach, this section explores the impact of varying the number of surrogate models on transferability. Table 2 presents the attack results using more surrogate models, including Res18T , DensT , Res18F , and ASTF , and compares these results with those of SVRE, AdaEA, and CWA-existing interdomain ensemble methods. While these existing methods improve the transferability in the time domain compared to MI, they show lower success rates in the frequency domain. In contrast, our AIE method achieves substantial transferability improvements in both the time and frequency domains. Additionally, our method has a lower complexity of O(6n) (n is audio size), compared to SVRE's O(Mn) (M=16 is inner iterations) and Adaea's O(k(k+1)n) (k is number of models).

4.4 Ablation Studies

We further conduct ablation experiments to evaluate the effectiveness of the adaptive domain weight adjustment (ADWA). Additionally, we also analyze the impact of the hyperparameter k and perturbation ϵ. The experiments are conducted on the UrbanSound8k dataset.

Adaptively Domain Weight Adjustment (ADWA)

ADWA is a crucial strategy for leveraging collective vulnerabilities to enhance and balance cross-domain transferability. As illustrated in Figure 6, after applying ADWA, the cosine similarity of cw and cs both tend to 1, and their probability densities almost coincide. This demonstrates that the dominance levels of time and frequency domain models in the attack process tend to balance.

Parameters k on Adjustment Function

We introduce the hyperparameter k that controls the slope of the function of η with respect to ρ in this paper. Figure 7a presents the effect of different k on the average attack success rate of the adversarial examples. When k = 3, we obtain the highest average success rate in both the time and frequency domains. Therefore, k = 3 is set in our experiments.

Perturbation Magnitude ϵ

The impact of perturbation magnitude ϵ on the attack success rates is illustrated in Figure 7b. We observe that a larger perturbation results in a higher attack success rate in both the time and frequency domains. To balance the performance and the imperceptibility [Chen et al., 2023a; Chen et al., 2023b], we set the ϵ to 0.01 in our experiments.

5 Conclusion

This work proposes the Adaptive Inter-domain Ensemble attack for generating adversarial audio examples with enhanced cross-domain transferability. AIE introduces an adaptive domain weight adjustment method that dynamically assigns weights to each domain, effectively addressing domain attack imbalance and significantly improving cross-domain transferability. Additionally, extending AIE to include model ensembles within each domain further enhances overall transferability. Extensive experiments on various datasets consistently demonstrate that our approach achieves higher attack success rates on time and frequency domain models than existing attack methods. Our work could shed light on the great potential of boosting cross-domain transferability through a better design of the adversarial attack methods and provide a reference for attacks on other signals (e.g., electromagnetic signals). We hope our work will inspire further in-depth investigations into cross-domain attacks.

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grants (52071111) and Hainan Provincial Natural Science Foundation of China (623CXTD394).