Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network

1 Introduction

With the development of Artificial Intelligence Generated Content (AIGC), more creative and realistic productions are coming into the limelight [Chen et al.2024b, Eskimez et al.2024,Liu et al.2024,Luo et al.2023,Kong et al.2025,Zhang et al.2025]. Recently, the malicious use of AIGC for low-cost audio partial forgery manipulation (PFM) has raised people's safety concerns [Cai et al.2024]. As shown in Figure 1 (a), a malicious attacker manipulates small portions of the real audio to alter the original meaning. It can easily evade the detection of utterance-level countermeasures [Jung et al.2022,Tak et al.2022,Wu et al.2024b] since the difference between these audio is small. To defend against PFMs, several datasets and advanced countermeasures [Cai and Li2024, Xie et al.2024] are proposed. Despite the remarkable progress achieved, some issues still exist.

∗Corresponding author.

1. Training Manner: Existing PFM countermeasures adopt a fully-supervised training manner [Wu et al.2024a,Cai et al.2022,Cai et al.2023], which requires fine-grained annotations, such as frame-level labels and ground-truth annotations depicted in Figure 1(b). However, in real-world scenarios, fine-grained annotations are costly and difficult to obtain. In this case, research on weakly-supervised temporal forgery localization (wATFL) is more popular and practical, yet such approaches remain underdeveloped.
2. Weak Supervision Challenge: Under a weaklysupervised training manner, it is highly challenging to locate subtle forgery regions based solely on utterance-level (i.e., binary) labels. Therefore, it is vital to develop countermeasures guided by the intrinsic principles of PFMs to prompt localization performance in weakly-supervised scenarios.

In this paper, we are motivated to introduce a weakly supervised learning localization method, namely progressive audio-language co-learning network (LOCO), which, to the best of our knowledge, is being explored for the first time in weakly-supervised partial forgery audio localization. Specifically, we first design the audio-language co-learning (A2LC) module to capture forgery consensus features by aligning semantics from temporal and global perspectives. In this module, a temporal forgery attention (TFA) adapter is used to model temporal forgery cues for semantic features. In addition, a prompt-enhanced forgery feature (PFF) adapter is devised to dynamically incorporate forgery-aware prompts into content semantics with valuable contextual and semantic cues from a global view. Through a co-learning manner, forgeryaware consensus features can be learned to capture semantic inconsistency cues raised from PFM. Moreover, due to the lack of fine-grained annotations, it is intuitive to obtain frame-level labels for free from a self-supervised perspective. Thus, we devise a progressive refinement strategy (PRS) that gradually guides the optimization of forgery-aware features through supervised semantic contrastive learning (SCL) to further amplify the semantic distinction between real and fake content.

To sum up, the main contributions of this paper are presented as follows:

• We propose a novel wATFL method named LOCO 1 , which promotes the learning of forgery-aware features in precise detection and localization performance through co-learning and self-learning manners.
• We specifically design an A2LC module to mine the semantic inconsistencies caused by PFM and introduce PRS to progressively amplify the semantic differences between real and forgery content.
• We conduct extensive experiments to demonstrate that the proposed LOCO performs favourably against the state-of-the-art methods on three public datasets.

2 RelatedWork

2.1 Temporal Forgery Localization

Temporal forgery localization aims to determine the precise timestamps of forgery manipulation segments within an audio or video. To advance the research, various datasets [Yi et al.2021, Zhang et al.2023a,Cai et al.2022,Cai et al.2024] have been proposed and can be divided into two categories based on whether semantic information is modified.

BA-TFD [Cai et al.2022] was the first framework designed for the temporal forgery localization task, and it adopts a 3DCNN backbone with a boundary matching network (BMN) for localization, where features are guided by contrastive learning, boundary matching, and frame classification. Subsequently, BA-TFD+ [Cai et al.2023] replaced the backbone with a multi-scale transformer and enhanced the localization module with a BSN++ network. Additionally, UMMAFormer [Zhang et al.2023b] introduced a novel temporal feature abnormal attention module, and a parallel crossattention feature pyramid network to enhance features from sequential multimedia data and adopted an ActionFormer decoder for localization. Moreover, CFPRF [Wu et al.2024a]

1 Code and pre-trained models are available at https://github.com/ ItzJuny/LOCO.

devised a frame-level detection network in the first stage to learn robust representations for better indicating rough forgery regions and employed a proposal refinement network in the second stage to produce fine-grained proposals.

Despite the success of fully-supervised ATFL methods, they require massive and consuming frame-level artificial annotations, which limits their practicality in real-world scenarios where fine-grained annotations are unavailable.

2.2 Weakly-Supervised Learning

Currently, wATFL methods have not been explored, with the closest being weakly-supervised temporal action localization (wTAL) and video anomaly detection (wVAD). Most of these works apply multi-instance learning (MIL) loss due to its ability to learn discriminative representations under weak labels. Based on the MIL loss, CoLA [Zhang et al.2021] also introduced a snippet contrast loss to refine the hard snippet representation in feature space, which guides the network to perceive precise temporal boundaries. CO2-Net [Hong et al.2021] devised a cross-modal consensus mechanism to capture inter-modality consistency and filter out task-irrelevant redundant information. CASE [Liu et al.2023] adopted unsupervised snippet clustering to explore the underlying structure among the snippets. SAL [Li et al.2025] designed a multilevel semantic learning branch and an adaptive actionness learning branch to introduce second-order video semantics and learn class-agnostic action information, respectively. To model anomaly semantics, PE-MIL [Chen et al.2024a] and LPCF [Pu et al.2024] adopted prompt-enhanced learning to detect various abnormal events while ensuring clear event boundaries. Unlike these end-to-end MIL methods, PMIL [Ren et al.2023] introduced an additional second stage to directly classify proposals based on temporal region of interest features. FuSTAL [Feng et al.2024] incorporated an EMA-based distillation teacher model to enhance localization performance at the second stage.

However, these methods are not easily applicable to the wATFL task due to several differences. First, the modality is mismatched, as most approaches extract single-modality video features for analysis. Second, the tasks are different. wVAD is a detection task, whereas wTAL requires predicting timestamps of forgery proposals. In addition, the core of wTAL is the Temporal Class Activation Sequence (T-CAS), which is specifically designed for multi-class action recognition, whereas wATFL focuses on modeling temporal forgery probability using attention mechanisms. Therefore, the action proposals derived from T-CAS and attention scores contain excessive redundancy, requiring hyperparameter adjustments to achieve satisfactory performance.

3 Methodology

3.1 ProblemDefinition

Assume we have N untrimmed training audio {Xn}Nn=1 . The goal of wATFL is to train the model using utterance-level labels Yn ∈ {0, 1}, where Yn = 1 indicates a partial forgery audio and Yn = 0 otherwise. During inference, the framelevel forgery scores {ˆ st}Tt=1 and a set of forgery proposals {(c, s, e)} are predicted for each audio, where T denote the number of frames, c, s and e represent the confidence score, the start timestamp and the end timestamp of each proposal, respectively.

3.2 Overview

Due to the differences between weakly-supervised tasks, we specifically design a progressive audio-language co-learning network for the wATFL task, with the structure shown in Figure 2. Based on the design principle of PFMs, this network aims to mine temporal forgery traces of semantic-driven modifications through co-learning and self-supervision manners to compensate for the lack of binary supervision.

First, the semantic information of partial forgery audio is deliberately modified through the pipeline shown in Figure 1, such as insertion, deletion, and replacement operations. To this end, an audio-language co-learning module is devised to learn discriminative features by mining semantic inconsistency, as detailed in Section 3.3. Specifically, given an input audio, an automatic speech recognition (ASR) model is used to extract the audio features with semantic information. These features are then passed to a temporal forgery attention adapter to extract temporal-enhanced forgery features.

Second, inspired by prompt-enhanced learning, the forgery-aware prompts are constructed using utterance-level annotations together with learnable prompts. Followed by a prompt-enhanced forgery feature adapter, semantic priors are incorporated into audio features dynamically to provide prompt-enhanced features with valuable contextual and semantic cues from a global view. In addition to using segmental MIL to guide these two discriminative features, a colearning loss is introduced to ensure the alignment of semantics and features.

Finally, since MIL often focuses on segmental-level predictions, it ignores the relationships between instances within a segment (e.g., semantic information). Thus, a progressive refinement strategy is designed to optimize forgery-aware features in a self-supervised manner by using semantic-based contrastive learning, as detailed in Section 3.4.

3.3 Audio-Language Co-learningModule

The A2LC module aims to mine the contextual semantic inconsistencies introduced by partial forgery manipulations, thereby improving the model's ability to detect forgery segments in partial forgery audio. Specifically, the A2LCmodule consists of a TFA adapter and a PFF adapter to collaboratively capture forgery-aware features, which are optimized through a co-learning strategy, as depicted in the left part of Figure 2.

Temporal Forgery Attention Adapter: Due to the principle behind semantical-driven manipulations, a pretrained audio-to-language (a.k.a, ASR) feature extractor, namely XLS-R-300M, is applied to extract initial audio features Fa ∈ RT×D with content semantics, where T denotes the number of frames in the audio and D is the dimension of features. Subsequently, to model temporal forgery cues, a TFA adapter is applied to enhance features through several residual blocks RB(·) with a temporal attention mechanism, and the latent forgery-aware features F t a ∈ RT×D′ are obtained as:

where ⊙ denotes the element-wise multiplication, αRB are learnable parameters, andD′ is the feature dimension. Subsequently, the frame-level scores ˆ St = FC(F t a , αtFC ) ∈ RT×2 are obtained by applying a classifier layer FC(·), where αtFC are learnable parameters.

Prompt-Enhanced Forgery Feature Adapter: Based on prompt-enhanced learning [Chen et al.2024a], forgery-aware prompts are introduced to provide context cues and semantic information for wATFL from a global view. Specifically, forgery-aware prompts are constructed by concatenating the category embedding for utterable-level class ci ∈ {real, fake}, with the learnable embedding that consists of l context tokens to form a complete sentence token, and thus, the input of text encoder for one class is presented as {e1, ..., el, ci}. The overall label prompt embedding Prompt ∈ RD′ is the CLS token output of a pre-trained text encoder [Devlin et al.2018]. With forgery-aware prompt and forgery features F t a in hand, prompt-enhanced forgery features F p a are generated by a feed-forward network (FFN) layer and a skip connection, presented as follows:

Such an implementation allows prompt-enhanced features F p a to capture the related forgery semantic context cues from audio. Similarly, the frame-level predictions ˆ Sp are obtained by applying a classifier layer to prompt-enhanced latent features.

Objective Function of Co-Learning: This optimization process is designed to guide two forgery features to learn complementary information from each other based on a colearning idea. Following the weakly-supervised methods, the MIL-based loss is first applied to optimize each forgery feature by selecting top-K frames with the highest confidence scores, which can be denoted as:

where K = 50, and v ∈ {t, p} denotes the predictions based on forgery features {F t a , F p a }. Then, the frame-level MIL loss is computed using a parameter-free loss function MSEpg(·), namely the P2sGrad-based mean squared error (P2sGrad-MSE) [Wang and Yamagishi2021], which alleviates the imbalance issue where true frames outnumber forgery frames, and can be presented as follows:

where Y is utterance-level label to indicate a partial spoof audio (Y = 1) or a real audio (Y = 0).

Additionally, a Kullback-Leibler (KL) divergence is utilized to ensure the alignment of semantics and feature distributions. Given the KL-divergence is not bounded, i.e., DKL ∈ {0,∞)}, taking the exponential of its negative value transforms the objective from maximization to minimization.

Then, the transformed objective is bounded within (0, 1], which is numerically advantageous. The loss for feature colearning can be denoted as:

3.4 Forgery LocalizationModule

The forgery localization module aims to generate forgery proposals for each audio. Unlike the wTAL localization module, this localization module computes a temporal forgery-class activation sequence (T-FAS) specifically tailored for wATFL tasks. Specifically, instead of using background-suppressed operations, attention scores, and T-CAS to generate proposals, we apply the softmax function to the output scores { ˆ St, ˆ Sp} along the second dimension. This operation ensures that the binary scores ('real' and 'fake') for each frame sum to 1, and can be denoted as follows:

where λa = 0.9 and the second class represents the T-FAS, denoted as A = {at}Tt=1 , as no proposals are needed for real segments. Each score in A indicates the probability of forgery for each frame. Instead of using a multiple thresholds strategy, we select forgery-aware threshold θf = 0.5 on the T-FAS to reduce redundancy proposals. Then, the forgery proposals P = {(cm, sm, em)}Mm=1 are generated by concatenating consecutive forgery frames, where the confidence score is computed as:

3.5 Progressive Refinement Strategy

Given that MIL focuses on segmental-level predictions, it is insufficient to model the relationships between audio frames (e.g., semantic information). The fundamental reason is the lack of fine-grained annotations for forgery feature optimization. Thus, based on a self-supervised idea, a progressive refinement strategy is designed to obtain frame-level labels for free. To ensure the accuracy of the pseudo frame-level labels and prevent the introduction of additional training noise, a two-stage process is applied for training LOCO.

In the first training phase, the base model is trained for 500k steps, with the aim of capturing semantic inconsistencies for forgery cues and the objective function is denoted as:

where λ1 is set to 0.1 to balance the training loss.

Then, in the second phase, pseudo frame-level labels are calculated using the start and end timestamps of proposals P = {cm, sm, em}Mm=1 predicted from the base model as:

where ¯ yt is the t-th frame in pseudo labels, M denotes the number of predicted proposals, and fps = 50 is the frames per second under a temporal resolution of 20 ms.

With these fine-grained labels, a semantic-based contrastive learning loss is designed to pull pairs of semantically similar audio frames in the feature space while pushing apart pairs with differing semantics, which can be presented as:

where SIM(·) is the cosine similarity between two frames, R denotes the number of frame pairs, Ir = 1 (or Ir = 0) indicates a similar (or dissimilar) frame in the r-th pair, fr and f + r (or f - r ) present the features of reference frame and similar (or dissimilar) frame, respectively.

Then, the objective function of the second stage is computed as:

where λ1 = 0.1 and λ2 = 0.01 are hyperparameters to balance the training loss. Finally, the pseudo frame-level labels are iteratively refined using predictions from the previous iteration, progressively improving the localization performance.

4 Experimental Results

4.1 Experimental Settings

Datasets. We evaluate our method on three datasets, including LAV-DF [Cai et al.2022], HAD [Yi et al.2021], and AV-Deepfake-1M [Cai et al.2024], where LAV-DF and HAD adopt a rule-based replacement to modify audio while AVDeepfake-1M is LLM-driven manipulation. Due to the lack of testing labels in the audio modality of AV-Deepfake-1M, we combine the training and validation sets to construct a new subset, ensuring non-overlapping speakers according to the original division strategy. Finally, we select 83,517, 30,212, and 20,417 audio for training, validation, and testing, respectively, according to the number and split ratio of other datasets.

Evaluation Metrics. We evaluate the proposed LOCO in terms of detection and localization performance. Following existing works, we use frame-level equal error rate (EER), accuracy (ACC), and area under the curve (AUC) to evaluate the detection task. For the localization task, we adopt average precision (AP) at the temporal intersection over union (IoU) thresholds [0.1 : 0.9 : 0.1], average recall (AR) with different average number of proposals (AN) {2, 5, 10, 20}.

Compared Methods. We select several fully-supervised start-of-the-art countermeasures, including detection methods (e.g., PSDL [Zhang et al.2023a], IFBDN [Cai and Li2024], FDN [Wu et al.2024a]) and localization methods (e.g., BA-TFD [Cai et al.2022], BA-TFD+ [Cai et al.2023], UMMAFormer [Zhang et al.2023b], CFPRF [Wu et al.2024a]).

Given the lack of wATFL methods, we adapt several wTAL methods (e.g., CoLA [Zhang et al.2021], CASE [Liu et al.2023], FuSTAL [Feng et al.2024], SAL [Li et al.2025]) for comparison with two key modifications. First, we replace their original input features with the XLS-R features used in our method to ensure a fair comparison. Second, we apply a softmax function to T-CAS and change the dimension to 2 to suit the wATFL task, while their original multithreshold strategy for proposal generation is retained. Additionally, we remove fine-grained supervision losses from the fully-supervised detection method and retain only utterancelevel losses to assess their detection performance under weak supervision scenarios.

4.2 Implement Detail

All experiments are conducted on a single GeForce RTX 3090 GPU. In terms of the model input, we preprocess each audio in a single-channel format with a 16-kHz sampling rate, then pre-trained XLS-R-300M [Babu et al.2021] and bertbase-uncased [Devlin et al.2018] are adopted to extract initial audio and textual features, respectively. For model training, we utilize the Adam optimizer with a batch size of 2 and a learning rate of 10-6 . The base model is trained for 300k steps on the HAD and LAV-DF datasets and 900k steps on the AV-Deepfake-1M dataset. Subsequently, the base model is trained for 15 epochs in the second stage and selected using the best results on the validation portion for testing.

4.3 Comparison with State-of-the-Art Methods

Frame-level Forgery Detection Results. Table 1 presents the frame-level detection results of the proposed LOCO compared with weakly-supervised and fully-supervised methods on three datasets. The results indicate that LOCO outperforms other state-of-the-art weakly-supervised methods, reducing EER values of 3.91%, 17.26% and 12.39% on the HAD, LAV-DF, and AV-Deepfake-1M datasets, respectively, compared to the second-best method. Additionally, we observe that while fully-supervised methods achieve significant performance, their performance drops drastically when only utterance-level annotations are used for training. For example, the best-performing model, denoted as FDN → FDN† , increases EER value from 0.08% to 8.47% on the HAD dataset. This may be due to these methods relying on finegrained label-based losses to guide complex modules for feature learning, making them unsuitable for direct adaptation to weakly-supervised settings. These results demonstrate the effectiveness of our proposed LOCO for the detection task.

Temporal Forgery Localization Results. Table 2 shows the temporal forgery localization comparison results on three datasets. From the results, we can see that the proposed LOCO outperforms the prior state-of-the-art weaklysupervised methods. Specifically, LOCO surpasses the previous best performance by 18.78% and 26.43% in terms of the mAP on the HAD and LAV-DF dataset, respectively. Additionally, our method achieves 18.89% mAP, representing a 12.92% improvement over the second-best method. This may be due to AV-Deepfake-1M containing more long-duration samples, which pose significant challenges for weakly-supervised performance, especially under high IoU thresholds. Moreover, even when compared to certain fully-supervised methods (e.g. BA-TFD and BA-TFD+), our model can achieve comparable results at low IoU (< 0.5) thresholds. The clear localization performance superiority demonstrates the effectiveness of the proposed LOCO again.

4.4 Ablation Studies

Impact of Different Components. As shown in Table 3, we investigate the contribution of the core components of the proposed LOCO. Specifically, we first conduct experiments with two training strategies for the TFA adapter (e.g., freezing and training the XLS-R feature extractor) to show whether the features can effectively adapt. And we observe that pretrained features are more suitable for wATFL because these features are derived from the large-scale and diverse audio corpus and have already captured rich semantic representations. In contrast, unfreezing and training the features may lead to disrupting the generalizable pretrained representations based on limited wATFL data. Additionally, replacing the TFA adapter with an FFN results in a 6.82% mAP decrease, highlighting the importance of audio features enhanced with temporal attention for effective localization. Moreover, we observe a 2.26% mAP decrease when the PFF adapter is removed, indicating that forgery-aware prompts are crucial for providing contextual semantic features. These results further demonstrate the effectiveness of the co-learning strategy in the A2LC module. Finally, benefiting from the learning of contrastive semantic features, the progressive refinement strategy improves the mAP values by 3.04%, respectively. All in all, each component of the LOCO serves a significant purpose and is thoughtfully designed.

Impact of Loss Functions. As shown in Table 4, we explore the contribution of each loss function in guiding forgery-aware feature learning. The MIL loss, serving as the foundational loss, enables temporal forgery features to achieve a localization performance of 68.79% mAP on the HAD dataset. When the P2sGrad-based MSE loss within MIL is replaced with the Binary CrossEntropy (BCE) loss, a reduction of 6.67% in mAP value is observed. This demonstrates that the P2sGrad-based loss is more suitable for the ATFL task, as it adaptively mitigates the bias of misclassifying fake frames as real without requiring the manual tuning of hyperparameters. With the addition of prompt-enhanced forgery features guided by the MIL loss, the localization performance improves by 2.86%. Furthermore, the inclusion of the KL-divergence loss to guide the co-learning of forgery features yields an additional 2.16% improvement in mAP performance. Finally, incorporating the semantic contrastive learning loss to enhance the distinction between different semantic content results in a further mAP increase of 3.04%.

Impact of Localization Thresholds. Table 5 shows the results at different forgery thresholds for generating localization proposals. As the threshold increases, the localization precision for forgery segments gradually decreases. The best mAP is achieved at a threshold of 0.5.

4.5 Qualitative Results

To gain more insight, the frame-level prediction scores predicted by our method are visualized in Figure 3. As demonstrated in Figure 3 (a), our method mitigates false alarms in

(b) 𝐿𝐿𝑀𝑀𝑀𝑀𝑀𝑀+ 𝐿𝐿𝐾𝐾𝐾𝐾

(c) 𝐿𝐿𝑀𝑀𝑀𝑀𝑀𝑀+ 𝐿𝐿𝐾𝐾𝐾𝐾 + 𝐿𝐿𝑆𝑆𝑆𝑆𝑆𝑆

Figure 5: t-SNE visualization of latent features with different loss functions. Green and red dots denote real and forgery frames.

normal audio. Figure 3(b) and Figure 3(c) exemplify the proficiency of our method in predicting precise forgery scores for long-term partial forgery audio with multi-segment and subtle manipulation segments. The ability to detect frame-level forgery content demonstrates the effectiveness of forgeryaware features based on semantic inconsistencies in capturing partial spoof manipulations.

Then, to intuitively demonstrate the localization performance compared to other SOTA methods, we show qualitative localization results in Figure 4. It is evident that LOCO produces more continuous and accurate forgery proposals, even in cases of multi-segment forgery (left part of Figure 4) and subtle forgery spanning long durations (right part of Figure 4). In contrast, the compared methods tend to generate multiple, disconnected, shorter forgery proposals.

Finally, to demonstrate the effect of co-learning loss and semantic contrastive learning, we employ t-SNE [Laurens and Geoffrey2008] to visualize the latent features from the intermediate layer. In Figure 5(a), we find that real and forgery frame-level features tend to be mixed at the boundaries when only audio features used. After incorporating prompt features for co-learning, this issue is alleviated, however, some forgery features remain mingled with the real ones, as shown in Figure 5(b). With the inclusion of the SCL loss, a more distinct separation emerges between real and forgery frames, as depicted in Figure 5(c). These findings demonstrate that colearning and the SCL loss effectively enhance the distinction between real and forgery frame-level features.

5 Conclusion

In this work, we present a novel method LOCO for wATFL, which aims to capture semantical inconsistencies and amplify the semantical difference between real and forgery content. Thus, an audio-language co-learning module is introduced to ensure the alignment of semantics and features from temporal and global perspectives. Then, a localization module is applied to generate proposals for each audio, where pseudo frame-level labels are used in the next stage of the progressive refinement strategy. In such a self-supervised manner, forgery-aware features are enriched to generate a clear forgery boundary. Extensive experiments show our method achieves SOTA performance on three public benchmarks. In the future, how to improve localization under a more precise IoU threshold (e.g., excessive 0.9) and long-duration samples, is a problem worthy of long-standing research.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.62441237, No.62261160653, No.62172435), the Guangdong Provincial Key Laboratory of Information Security Technology (No. 2023B1212060026), and Henan Province Innovation Scientists and Technicians Troop Construction Projects of China (No.254000510007).