Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

1 Introduction

Neuroimage studies [Joassin et al., 2011] suggest that the interactions between voice and face are not only at a perceptual level, but also in the person recognition process. Intuitively, when hearing a person's voice, even if we don't see the person, we immediately have its appearance in our minds. Even for a stranger we have never seen before, we could 'construct' what he looks like through his voice. While, whether voice and face are relevant and whether they can be perceived by Deviate Positives the computational models? In recent years, researchers made a sustainable effort on an enduing neural network the ability to recognize faces with the same identity only by hearing the voice or recognizing the voice from the face. It could be formulated as a voice-face association problem. The problem is an interesting scientific study on its own, and it could also benefit practical applications such as virtual reality, criminal investigation, multimodal information retrieval, etc.

∗Corresponding author

Figure 1: False Negatives and Deviate Positives examples in Crossmodal Instance Discrimination. (1) Without identity labels, negative pairs from different videos could actually belong to the same identity, such as the anchor face and negative voice in red border, which leads to false negatives. (2) The voice-face correlation might be personalized or irrelevant caused by the noise in the videos, which leads to deviate positives. The voice modality is displayed by marking a trumpet symbol in the image.

Psychological and behavioral studies show that voice and face integration skills could emerge very early in our life by processing a number of auditory-visual examples [Bahrick et al., 2005], without intended discipline. This indicates that our computational models may be feasible to learn the association between voice and face in an unsupervised manner. More recently, contrastive learning has become a prominent tool in unsupervised representation learning, leading to state-of-theart results in many fields such as vision, language, and speech. A rock line of papers [Morgado et al., 2021b; Zolfaghari et al., 2021; Morgado et al., 2021a; Alayrac et al., 2020] have employed the Cross-modal Instances Discrimination (CID) method to learn the multi-modal representations.

In the CID methods, the representations are learned by two opposite forces, the repellence of negative pairs and the attraction of positive pairs. The positive pairs are generated by the audio and images extracted from the same video, and the negative pairs are those extracted from different videos. Then, a contrastive loss (e.g. InfoNCE [Oord et al., 2018]) is used to pull close the positive pairs and push apart the negative pairs. Despite their improving performance, the CID methods are still confronted with two challenges: (1) False negative problem. The negative pairs are sampled from different videos regardless of their semantics. For example, two videos have the same identity (person), the representations of the voice and face should be pulled close, but they have undesirably pushed apart because they are sampled as negative pairs, as in Figure 1. Since the current CID methods need to utilize a large number of negatives, the problem of the false negative is non-ignorable. (2) Deviate positive problem. Previous methods have learned the representations by encouraging the network to memorize all the voice-face associations in the dataset. While they ignore the biometric specificity across different identities. Some identities hold an 'untypical' or personal voice face correspondence. Moreover, a large amount of video data inevitably contain noise, including blocked faces, excessive head rotation, background music, the voice from outside of the screen, etc. We named them 'deviate positives' instead of 'false positives', because they are all reasonable existing, and to distinguish from supervised learning where there are ground-truth labels.

We firstly reveal the impact of the false negatives and deviate positives in Section 3.1 from a theoretical view. To combat the above issues, we then propose a Cross-modal Prototype Contrastive Learning (CMPC) algorithm, which takes advantage of previous contrastive methods meanwhile resisting adverse effects of false negatives and deviate positives. Specifically, we obtain the semantic structure by clustering on each modality, and each cluster centroid is used as the prototype of the corresponding instance. The learned prototypes are used to construct the semantic positives with the instance in another modality. We use an online clustering approach, which keeps the consistency of instance representations and reduces computation. Additionally, we dynamically downweight the deviate positives' contribution to overall loss by an Instance Recalibrate method. By combining our methods, we can outperform previous state-of-the-art unsupervised methods on various voice-face association evaluation protocols. Moreover, with low-shot supervision, our method achieves significant improvement compared to previous instance-wise contrastive learning, even outperforms fully supervised models.

The rest of this paper is organized as follows. We first introduce the related work in the following Section 2 , then we analyze the problems and present the details of the proposed approach in Section 3. The experiments are given in the Section 4. Finally, we conclude our paper in Section 6.

2 RelatedWork

2.1 Learning Association of Voice and Face

In recent years, the interests for the human voice-face crossmodal learning tasks have surged, and the attempts have shown the feasibility of recognizing a person's face from his voice or identifying a speaker only by giving his face image. SVHF [Nagrani et al., 2018b] firstly raised voice-face matching and retrieval task. They concatenate the feature from different modalities and then learn a N-way classification task by minimizing a cross-entropy loss on the personal identity label. Instead of explicitly learning the joint embedding between voice and face, DIMNet [Wen et al., 2019a] learns a shared representation under supervision from one or more covariates, such as gender, ethnicity, and age, by a multi-task object function. [Horiguchi et al., 2018] and [Kim et al., 2018] use N-pair loss and triplet loss to increase the voiceface similarity of the same person respectively. In [Chung et al., 2020], they focus on the audio-visual speech separation task by using the still face images rather than lip movement on video clips. [Nawaz et al., 2021] tries to explore the language dependent problem in voice-face association. For example, the question about: "Can a speaker be recognized irrespective of the spoken language?" In [Wen et al., 2021], they used a two-level inter-modality and intra-modality alignment.

Also, they propose a reweighting scheme to better handle the hard samples. In [Nagrani et al., 2018a], they learn the crossmodal representation in a self-supervised manner by the contrastive loss, and a curriculum learning schedule boosts the performance. Other works seeks the possibility of generating the human face from voice. [Wen et al., 2019b] uses a generative adversarial network to reconstruct someone's face. [Choi et al., 2020] links the inference stage and generation stage by a conditional generative adversarial network.

2.2 Contrastive Learning

Contrastive learning has recently become a prominent tool in unsupervised representation learning, leading to state-of-theart results in many fields such as vision, language and audio. Contrastive learning either aims to predict parts of the data, or to discriminate between individual instances. The former explicitly define a proxy tasks, such as colorization, jigsaw solving, temporal coherence and future prediction [Han et al., 2020; Pan et al., 2021]. The later, instances discrimination can be formed as a N-classification task which N is the number of instances. SupCon [Khosla et al., 2020] extend the self-supervised contrastive approach to the fullysupervised setting, allowing us to effectively leverage label information. The main difference between different contrastive learning approaches lies in their strategy for definition of positive pairs. For example, image random cropping and rotation, or different augmentation of the same instance. After getting the positive pairs, comparing with as many negative examples is important to boost the performance. Other works focus on the hard negatives mining [Kalantidis et al., 2020] or unbiased negatives sampling [Chuang et al., 2020; Wu et al., 2021] in contrastive learning.

Clustering-based contrastive learning [Caron et al., 2018; Caron et al., 2020; Zhan et al., 2020] has been proposed to learn between instance embeddings and cluster assignments. Among them, PCL [Li et al., 2021] formulates the clustering and representation learning as an Expectation-Maximization (EM) algorithm, which approximates and maximizes the loglikelihood function. While, whether we can learn effective representations if prototypes and instance vectors are derived from different modalities will be explored in this paper. Closer to our work, XDC [Alwassel et al., 2020] performs iterative clustering and classification to learn cross-modal representation. The cluster assignments of one modality are considered as pseudo-label of another modality, following a classification objective. While, the cross-entropy classification loss is sensitive to the noisy label, the inaccurate pseudo-label make it difficult for convergence. However, our method is conceptually different from the XDC. In XDC, the cluster assignments of one modality are considered as pseudo-label of another modality, following a classification objective, which results in two weaknesses: (1) The cross-entropy classification loss is sensitive to the noisy label, the inaccurate pseudolabel make it difficult for convergence. (2) The initialization of pseudo-label gives endow the semantic information to the cluster centroid, which may lead to training bias. In our method, we actively avoid the pseudo-label and noisy label problem through the clustering and contrast iteration.

3 Methodology

3.1 Preliminary

Consider a dataset of N video instances X = {(xVi , xFi ) | i = 1, ...,N} where each instance have two modalities, i.e., voice and face. The goal of voice-face association is to jointly learn high-level representations (vi, fi) for both modalities to capture cross-modal correlations. The voice and the face of the same identity (person) are expected to have similar representations and vice versa. To this end, let gM : xM → Rdm be a parametrized modality specific backbone neural network that takes xM as input from modality M ∈ {V, F} and outputs a representation of dM dimension. All the representations are normalized onto a unit hypersphere. In this work, we follow the unsupervised setting in [Nagrani et al., 2018a] to train the encoders using only video instances without the exact identity labels.

State-of-the-art unsupervised audio-visual representation learning methods [Alayrac et al., 2020; Morgado et al., 2021b; Patrick et al., 2021] usually follow the paradigm of Cross-modal Instance Discrimination (CID). For a batch of different video instances, CID uses the voice-face pairs from the same instance as the positive pairs and those from different instances as the negative pairs. The representations are then trained in a contrastive learning manner [He et al., 2020; Chen et al., 2020] across the two modalities. For simplicity, we only describe the case from voice to face and the same goes for the other side. Given a positive pair (vi, fi), CID establishes a negative face representation set NFi = {fj | j ̸= i} and seeks to maximize the mutual agreement between vi and fi by minimizing the InfoNCE loss [Oord et al., 2018]:

where τ is the temperature hyper-parameter. This process can be interpreted as maximizing the similarity spi = vT i fi between the positive pair while suppressing the similarity sni,j = vT i fj between the negative pairs. Moreover, the gradients of spi and sni,k are computed as:

As mentioned in Section 1, CID suffers from the false negative and the deviate positive problems. On one hand, for a false negative (vi, fk), their similarity sni,k could be high as they share the same identity. From Eq. 2, this will impose a heavy penalty which pushes vi away from fk. This is suboptimal as vi and fk should be matched together. On the other hand, for a deviate positive, the positive similarity spi tends to be small. According to Eq. 3, there will be a heavy penalty which pulls the deviate positive close, which could lead to local optima and hamper the generalizability of the representations.

3.2 Cross-modal Prototype Contrast

To tackle false negative problem, we propose a cross-modal prototype-based contrastive learning method. Our motivation is to extract a set of prototypes which encodes the underlying semantic structure of the whole dataset by clustering the video instances and contrast the instances with the prototypes. For each instance, its representation in one modality is pulled close to the prototype where it belongs in the other modality and pushed away from other prototypes. As false negatives usually belong to the same prototype, they will be optimized to have similar representations.

Following [Li et al., 2021], we conduct clustering and contrast iteratively: the prototypes are reconstructed after each training epoch and used in the next epoch. However, as the encoders change rapidly during training, directly gathering the representations from different training batches induces severe representation inconsistency, which harms the quality of the prototypes. For this reason, we design instance feature memories (MemVi ,MemFi ) for each video instance xi. The instance feature memories are defined as the momentum update of the instance representations in both modalities:

where m ∈ [0, 1) is a momentum coefficient. Compared with (vi, fi), the change of (MemVi ,MemFi ) are much smoother during training, which provides better consistency. After each training epoch, we cluster all instances into K distinct groups in each modality based on their instance feature memories. Here we simply adopt the k-means algorithm but we note that our method is agnostic to clustering methods. The clustering centroids C V = {cVi | i = 1, ...,K} and C F = {cFi | i = 1, ...,K} are used as the prototypes.

Next, we apply the contrastive loss between the instance representations and the semantic prototypes in a cross-modal way. For each instance representation (vi, fi), we pull vi close to the face prototypes cFs where fi belongs and push it away from all other face prototypes:

The same computation goes for ℓp(fi, C V ). To ensure robust semantic prototype estimation, we cluster the instance feature memories for R times with different numbers of clusters and aggregate the instance-wise contrastive loss to retain local smoothness. The overall CMPC loss is defined as:

Compared to previous clustering-based contrastive learning methods [Caron et al., 2018; Li et al., 2021], our CMPC differs in the following two aspects: (1) They adopt a offline update strategy where the representations of all training instances are recomputed before clustering to achieve representation consistency. This induces heavy computation cost, especially in large-scale datasets. On the contrary, our method leverages an online strategy which achieves representation consistency without apparent computation overhead. (2) Our method is designed for cross-modal contrastive learning, while these methods mainly focus on one modality.

3.3 Instance Recalibration

To address the deviate positive problem, we propose a Instance Recalibration mechanism to reduce the importance of deviate positives on representation learning.

For a video instance, it is more likely to be deviate if (1) the similarity of its voice representation vi and face representation fi is small, and (2) the similarity of its corresponding voice prototype cVs1 and face prototype cFs2 is large. Base on the above observations, we design a deviation function ρ(·) which measures the possibility of an instance being deviate:

The deviation function describes how far the instance representation similarity deviates from prototype similarity. The smaller ρ(i) is, the more likely the i-th instance is a deviate positive. We then compute a recalibration factor wi by:

Inspired by [Morgado et al., 2021a], we use the cumulative distribution function (CDF) to control the shape of recalibration function. FN is the CDF of a Gaussian distribution N(µ + δσ, κσ2 ), where µ and σ are the mean and the variance of the deviation function ρ(i) on all instances, and δ and κ are hyper-parameters to control the shape of the recalibration function. The recalibration factor wi is then use to rescale the CMPC loss of each instance. The final loss function are defined as:

4 Experiments

4.1 Dataset

We use the VoxCeleb dataset [Nagraniy et al., 2017] in our experiments, which contains 21,063 video clips of 1251 celebrities, extracted from videos uploaded to YouTube. The dataset is gender-balanced and speakers span a range of different ethnicity, accent, profession, and age. Following the previous works [Nagrani et al., 2018a; Wen et al., 2021], random 1001 identities are randomly sampled as training data and the rest 250 identities as the test set. The face images are generated by facial bounding boxes cropping the video frames. The voice clips are extracted from the soundtrack of the videos with a 16kHz sampling rate at 16-bit PCM.

The test set is annotated with different demographic attributes to construct 5 testing groups, including the unstratified group (U), stratified by gender (G), stratified by nationality (N), stratified by age (A), stratified by gender and nationality (GN), and stratified by (G,N,A). The demographic groups allow us to control the important attributes in the experiments. For example, in the G-group, all the alternative instances have the same gender. The matching and verification tasks are conducted under different demographic partitions. All the evaluation lists are released.

5 Evaluation Protocols

To appraise our learned representations, following protocols are conducted for evaluation:

Matching. We are given a probe from one modality, and two alternatives from another modality, including one that belongs to the same person as the probe, and another imposter that comes from another person. The task is to affirm which alternative matches the probe. We report the bidirectional matching performance, namely voice to face (V-F) matching accuracy and face to voice (F-V) matching accuracy.

Verification. We are given a voice clip and face image. The task is to judge whether they belong to the same identity. The basis of judgment is to compare the similarity of two input features with a given threshold. With different thresholds, we could get various false positive rates and true positive rates. The final performance is reported by the Area Under the ROC curve (AUC).

Retrieval. Given a probe from one modality, and a large size gallery which contains candidates from another modality. One or more candidates might match the probe. The task is to order the gallery such that the matched candidates lie at the top of the ordering. We report the bidirectional performance in terms of Mean Average Precision (mAP).

5.1 Implementation Details

Voice and face preprocessing. We randomly crop a 5 seconds length speech and extract the commonly used logmel feature which is the mel-spectrogram at log-scale during training. The logmel features are transformed with 100ms windows width, 10ms frame shift and 64 mel-bins, yielding an input at size of 64 × 500. No further speech-specific preprocessing (e.g. noise suppression, voice activity detection) is applied. The face frames are extracted at 1fps from the video stream and cropped surrounding a human face. The input is a series of RGB images resized at 224 × 224 pixels. The only augmentation used on the face frames is a horizontal flip.

Network architecture. The ResNet-34 [He et al., 2016] is adopted as the voice and face encoder network. Our network is trained from scratch without any pretrained data. The original fully-connected layer of each encoder is removed. The weights of the two encoders are not shared, while they are trained jointly by the approach introduced above. We get the features after the last convolutional layer as the representations, and they are average pooled to the 512-dimensions. For both the voice and face representations, they are projected normalized into the unit sphere by L2 norm. In the evaluation phase, the projection network is removed, and only calculates the similarity between learned representations.

Optimization. We use the Adam optimizer, with a momentum of 0.9, a weight decay of 0.002 and a batch size of 128. We train for 3.2M iterations. We use a cosine annealing schedule with warm-up to tune our learning rate. The initial learning rate is 1e-4 and increases linearly to 5e-3 in the first 0.3M iterations, then gradually decreases to 1e-4. We also warmup our network in these 0.3M iterations using CID loss. The momentum coefficient in computing instance feature memory is 0.5. We set temperature τ to 0.03, number of clusters K = {500, 1000, 1500}. The σ and κ in the recalibration are set to -1 and 0.1 respectively. We use KeOps

[Charlier et al., 2021] for the efficient k-means clustering. Our code, model and testing resources are released 1 .

5.2 Quantitative Results

We now compare our CMPC to prior works, including supervised and unsupervised methods on various voice-face association evaluation protocols. we can outperform previous state-of-the-art unsupervised methods on various voice-face association evaluation protocols.

Table 1 summarizes the cross-modal matching performance, and Table 2 presents the cross-modal verification and retrieval results. We could see that our CMPC outperforms the previous unsupervised approaches by a large margin, bridging the gap between unsupervised and supervised methods. As this task is challenging, even human performance is around 80% [Nagrani et al., 2018b] on the matching protocol. Our results indicate that it is feasible to learn robust voice-face representation through unlabeled videos.

Compared methods. Among the compared methods, DIMNet [Wen et al., 2019a] uses one or more covariates, such as gender and identity, by a multi-task object function. Due to the use of more supervised information, it has achieved good results. Wen's [Wen et al., 2021] adaptively re-weights identity samples, while its re-weighting method is based on the identity label which is not used in our method. Meanwhile, their face data are obtained from the VGGFace dataset, which is different from the previous and our method. The results we report are reproducing 2 their method on the VoxCeleb dataset. PINs [Nagrani et al., 2018a] also employs a contrastive loss to learn the representation under an unsupervised setting, while our contrastive loss is distinct from theirs in which a positive pair only compares with one negative pair at once. It can be seen as a variant of the triplet loss.

1 https://github.com/Cocoxili/CMPC

Low-shot learning. To further evaluate the learned representations, we conduct our experiments on the low-shot training, which only uses a few instances per-identity. These instances are labeled by the identity information. We follow the DIMNet [Wen et al., 2019a] setup to finetune our network, while we do not use other covariates such as gender, nationality, etc. We representatively select 5 instances per-identity, which is about one-third of the whole dataset. We report the evaluation results in Table 1 and Table 2. Due to the better voice and face representations we learned, the results even outperform those with entire dataset supervision.

Ablation Study. To shed light on the contributions of our two components, cross-modal prototype contrast and instance recalibration, we report the matching and verification results on the unseen-unheard unstratified group (U) of VoxCeleb dataset as in Table 3. The other experimental settings are kept identical. Both two components yield a positive impact on the results, individually and jointly.

5.3 Qualitative Analyses

In Figure 4, we visualize the learned representations of voice clips and face images using t-SNE. We color the sample points by identity labels and gender labels respectively. We compare our approach with the CID method [Morgado et al., 2021b] and fully supervised approach [Wen et al., 2019a].

The CID only focuses on the local association between instances and ignores the semantic structure. It undesirable pushes false-negative instances apart. Therefore, different instances representations of the same person will appear far away, such as the samples marked in the red circles. While, in the supervised method, every identity's representations are brought together tightly. In our method, we keep both voice and face representations of the same identity closer together even without using any labels, due to the cross-modal prototype contrast. Another interesting thing, the representations are not well separated by gender even in the supervised method, as we do not use gender supervision. While in our approaches, they are separated naturally.

6 Conclusion

In this paper, we propose a novel unsupervised approach to learn voice-face association from the talking-face videos. We use the natural relevance between audio and visual frames to establish a self-supervised approach. Our method is based on the framework of contrastive learning, where the selection of positive and negative samples is crucial. We reveal their adverse impact by analyzing their gradient flow in the training phase. To tackle the false negative problem, we propose a cross-modal prototype contrast algorithm by estimating the semantic prototypes of each instance, then we replace the instance-to-instance contrast with instance-to-prototype contrast in a cross-modal way. To tackle the deviate positive problem, we dynamically recalibrate the instances by a deviation function. Experiments on cross-modal matching, verification, and retrieval have shown the superiority of our learn representations.

2 https://github.com/KID-7391/seeking-the-shape-of-sound

Acknowledgments

This work is supported by the major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" project 2020AAA0104803.