Multi-Prototype Networks for Unconstrained Set-based Face Recognition

1 Introduction

Recent advances in deep learning have remarkably boosted the performance of face recognition. Some approaches claim to have achieved [Taigman et al., 2014; Li et al., 2016; Zhao et al., 2017; Zhao et al., 2018a] or even surpassed [Schroff et al., 2015; Wang et al., 2018b; Zhao et al., 2018b] human performance on several benchmarks. However, these approaches only recognize faces over a single image or video sequence. Such scenarios deviate from the reality. In practical face recognition systems (and arguably human cortex for face recognition), each subject face to recognize is often enrolled with a set of images and videos captured with varying conditions and acquisition methods. Intuitively such rich information can benefit face recognition performance, which however has not been effectively exploited.

∗ Jian Zhao is the corresponding author. Homepage: https: //zhaoj9014.github.io. This work was done during Jian Zhao served as a short-term "Texpert" research scientist at Tencent FiT DeepSea AI Lab, Shenzhen, China.

In this paper, we address the challenging task of unconstrained set-based face recognition, which was firstly introduced in [Klare et al., 2015] and is more consistent with realworld scenarios. In this task, the minimal facial representation unit is defined as a set of images and videos instead of a single medium. It requires solving a more difficult set-to-set matching problem, where both the probe and gallery are sets of face media. This task raises the necessity to build subjectspecific face models for each subject individually, instead of relying on a single multi-class recognition model as before. An illustration is given in Fig. 1 to show the difference between traditional face recognition over a single input image and the targeted face recognition over a set of unconstrained images/videos. The most significant challenge in the unconstrained set-based face recognition task is how to learn good representations for the media set, even in presence of large intra-set variance of real-world subject faces caused by varying conditions in illumination, sensor, compression, etc., and subject attributes such as facial pose, expression and occlusion. Solving this problem needs to address these distracting factors effectively and learn set-level discriminative face representation.

Recently, several set-based face recognition methods have been proposed [Chen et al., 2016; Chowdhury et al., 2016; Hassner et al., 2016]. They generally adopt the following two strategies to obtain set-level face representation. One is to learn a set of image-level face representations from each face medium in the set individually [Chowdhury et al., 2016; Masi et al., 2016], and use all the information for following face recognition. Such a strategy is obviously computationally expensive as it needs to perform exhaustive pairwise matching and is fragile to any outlier medium captured under unusual conditions. The other strategy is to aggregate face representations within the set through simple average or max pooling and generate single representation for each set [Chen et al., 2016; Sankaranarayanan et al., 2016]. However, this obviously suffers from information loss and insufficient exploitation of the image/video set.

To overcome the limitations of existing methods, we propose a novel Multi-Prototype Network (MPNet) model. To learn better set-level representations, MPNet uses a Dense SubGraph (DSG) learning sub-net to implicitly factorize each face media set of a particular subject into a number of disentangled sub-sets, instead of handcrafting the set partition using some intuitive features. Each dense subgraph corresponds to a sub-set (representing a prototype) of face media that are with small intra-set variance but discriminative from other subject faces. MPNet learns to enhance the compactness of the prototypes as well as their coverage of large variance for a single subject face, through which heterogeneous attributes within each face media set are sufficiently considered and flexibly untangled. This significantly helps improve the unconstrained set-based face recognition performance by providing multiple comprehensive and succinct face representations, reducing the impact of media inconsistency. Compared with existing set-based face recognition methods [Chen et al., 2016; Chowdhury et al., 2016; Hassner et al., 2016], MPNet effectively addresses the large variance challenge and offers more discriminative and flexible face representations with lower computational complexity. Also, superior to naive average or max pooling of face features, MPNet effectively preserves the necessary information through the DSG learning for set-based face recognition. Main contributions of this work can be summarized as below:

• We propose a novel and effective multi-prototype discriminative learning architecture MPNet. To our best knowledge, MPNet is the first end-to-end trainable model that adaptively learns multiple prototype face representations from sets of media. It can effectively address the large intra-set variance issue that is critical to set-based face recognition.
• A Dense SubGraph (DSG) learning sub-net is used in MPNet to automatically factorize each face media set into a number of disentangled prototypes representing consistent face media with sufficient discriminativeness. Through the DSG sub-net, MPNet is capable of untangling inconsistent media and dealing with faces captured under challenging conditions robustly.
• DSG provides a general loss that encourages compactness around multiple discovered centers with strong discrimination. It offers a new and systematic approach for large variance object recognition in the real world.

With the above technical contributions, our proposed model gains high performance for unconstrained set-based face recognition. It achieves currently best results on IJB-A [Klare et al., 2015] benchmark dataset with significant improvements over state-of-the-arts.

2 RelatedWork

Recent top performing approaches for face recognition often rely on deep CNNs with advanced architectures. For instance, the VGGface model [Parkhi et al., 2015; Cao et al., 2018], as an application of the VGG architecture [Simonyan and Zisserman, 2014], provides state-of-the-art performance. The DeepFace model [Taigman et al., 2014] also uses a deep CNN coupled with 3D alignment. FaceNet [Schroff et al., 2015] utilizes the inception deep CNN architecture for unconstrained face recognition. DeepID2+ [Sun et al., 2015b] and DeepID3 [Sun et al., 2015a] extend the FaceNet model by including joint Bayesian metric learning and multi-task learning, yielding better unconstrained face recognition performance. SphereFace [Liu et al., 2017], CosFace [Wang et al., 2018b], AM-Softmax [Wang et al., 2018a] and ArcFace [Deng et al., 2018] exploit margin-based representation learning to achieve small intra-class distance and large interclass distance. These methods enhance the overall performance via carefully designed architectures, which are however not tailored for unconstrained set-based face recognition.

With the introduction of IJB-A benchmark [Klare et al., 2015] by NIST in 2015, the problem of unconstrained setbased face recognition attracts increasing attention. Recent solutions to this problem are also based on deep architectures, which are leading approaches on LFW [Huang et al., 2007] and YTF [Wolf et al., 2011]. Among them, B-CNN [Chowdhury et al., 2016] applies a new Bilinear CNN (B-CNN) architecture for face identification. Pooling Faces [Hassner et al., 2016] aligns faces in 3D and partitions them according to facial and imaging properties. PAMs [Masi et al., 2016] handles pose variability by learning Pose-Aware Models (PAMs) for frontal, half-profile and full-profile poses to perform face recognition in the wild. These methods often employ separate processing steps without considering the modality variance within one set of face media and underlying multiple prototype structures. Therefore, much useful information may be lost, leading to inferior performance.

Our proposed MPNet shares a similar idea as subcategoryaware object classification [Dong et al., 2013] that considers intra-class variance in building object classifiers, and [Das and Lee, 2018a; Das and Lee, 2018b] that consider to construct a graph-matching metric for domain adaption. Our method differs from it in following aspects: 1) the "prototype" is not pre-defined in MPNet; 2) MPNet is based on deep learning and can be end-to-end trainable. It is also interesting to investigate the application of our MPNet architecture in generic object recognition tasks.

3 Multi-Prototype Networks

Fig. 2 visualizes the architecture of the MPNet, which takes a pair of face media sets as input and outputs a matching result for unconstrained set-based face recognition. It adopts a modern deep siamese CNN architecture for set-based facial representation learning, and uses a new DSG sub-net for learning the multi-prototype that models various representative faces under different conditions for the input. MPNet is end-to-end trainable by minimizing the ranking loss and a new DSG loss. We now present each component in detail.

3.1 Set-based Facial Representation Learning

Different from face recognition over a single image, the task of set-based face recognition aims to accept or reject the claimed identity of a subject represented by a face media set containing both images and videos. Performance is assessed using two metrics: percentage of false accepts and that of false rejects. A good model should optimize both metrics simultaneously. MPNet is designed to nonlinearly map the raw sets of faces to multiple prototypes in a low dimensional space such that the distance between these prototypes is small if the sets belong to the same subject, and large otherwise. The similarity metric learning is achieved by training MPNet with two identical CNN branches that share weights. MPNet handles inputs in a pair-wise, set-to-set way so that it explicitly organizes the face media in a way favorable to set-based face recognition.

MPNet learns face representations at multi-scale for gaining strengthened robustness to scale variance in real-world faces. Specifically, for each medium within a face media set, a multi-scale pyramid is constructed by resizing the image or video frame to four different scales. To handle the error of face detection, MPNet performs random cropping to collect local and global patches from each scale of the multi-scale pyramid with a fixed size, as illustrated in supplementary materials 1 App. A. To mitigate the imbalance of realistic face data (e.g., some subjects are enrolled with scarce media from limited images while some with redundant media from reduplicative video frames), the data distribution within each set is adjusted by resampling. In particular, the set containing scarce media (i.e., less than an empirically pre-defined parameter R) is augmented by duplicating and flipping images, which is intuitively beneficial with more relevant information. The large set with redundant media (i.e., more than R) is subsampled to the size of R. The resulting input streams to MPNet are tuples of face media set pairs and the associated ground truth annotations {(Xp1 ,Xp2 , yp )}, where Xp1 and Xp2 denote the two sets of the p-th pair, and yp denotes the binary pair-wise label.

The proposed MPNet adopts a siamese CNN architecture in which two branches share weights for pairwise setbased facial representation learning. Each branch is initiated with VGGface [Parkhi et al., 2015], including 13 convolutional layers, 5 pooling layers and 2 fully-connected layers. We make the following careful architectural design for each branch to ensure that the learned deep facial representations are more suitable for multi-prototype representation learning. 1) For activation functions, instead of using ReLU to suppress all the negative responses, we adopt the PReLU to allow negative responses. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk, benefiting convergence of MPNet. 2) We adopt two local normalization layers after the 2nd convolutional layer and the 4th convolutional layer, respectively. The local normalization tends to uniformize the mean and variance of a feature map around a local neighborhood. This is especially useful for correcting non-uniform illumination or shading artifacts. 3) We adopt an average operator for the last pooling layer and a max operator for the previous 4 pooling layers to generate compact and discriminative deep set-based facial representations. Note that our approach is not restricted to the CNN module used, and can also be generalized to other state-of-the-art architectures for performance boosting.

The learned deep facial representation for each face media set is denoted as {f1, f2, · · · , fR}. Here recall R is the specified size of the face media set after distribution balance.

3.2 Multi-Prototype Discriminative Learning

Throughout this paper, a prototype is defined as a collection of similar face media that are representative for a subject face under certain conditions. Face media forming the same prototype have small variance and one can safely extract representation by pooling approaches without worrying about information loss.

To address the critical large variance issue in set-based face recognition, we propose the multi-prototype discriminative learning. With this component, each face media set is implicitly factorized into a certain number of prototypes. Multi-prototype learning encourages the output facial representations to be compact w.r.t. certain prototypes and meanwhile discriminative for different subjects. Thus, MPNet is capable of modelling the prototype-level interactions effectively while addressing the large variance and false matching caused by untypical faces. MPNet dynamically offers an optimal trade-off between facial information preserving and computation cost. It does not require exhaustive matching across all of the possible pairs from two sets for accurately recognizing faces in the wild. It learns multiple prototypes through a dense subgraph learning as detailed below.

1 https://github.com/IJCAI19-ID110/Supplementary-Materials/ blob/master/ijcai19-supp.pdf

To discover the underlying multiple prototypes of each face media set instead of handcrafting the set partition, we propose a novel DSG learning approach. DSG formulates the similarity of face media within a set through a graph and discovers the prototype by mining the dense subgraphs. Each subgraph has high internal similarity and small similarity to the outside media. Compared with clustering-based data partition, DSG is advantageous in terms of flexiblity and robustness to outliers. Each subgraph provides a prototype for the input subject faces. We then perform face recognition at the prototype level, which is concise and also sufficiently informative.

Given a latent affinity graph characterizing the relations among entities (face media here), denoted as G=(V, E), each dense subgraph refers to a prototype of the vertices (V) with larger internal connection (E) affinity than other candidates. In this work, learning DSG within each face media set implicitly discovers consistent faces sharing similar conditions such as age, pose, expression, illumination and media modality, through which heterogeneous factors are flexibly untangled.

Formally, suppose the graph G is associated with an affinity A, and each element of A encodes the similarity between two face media: aij = aff(fi, fj). Let K be the number of prototypes (or equivalently, number of dense subgraphs) and Z = [z1, . . . , zK] ∈ Rn×K be the partition indicator: zik = 1 indicates the i-th medium is in the k-th prototype. The DSG aims to find the representative subgraph via optimizing Z through

where 1 is an all-1 vector. Here the 2nd constraint guarantees that every medium will be allocated to only one prototype. The allocation after learning would maximize the intra-prototype media similarity. This is significantly different from k-means clustering which needs to learn the centers and defines similarity based on the distance to the center.

This problem is not easy to solve. We therefore carefully design the DSG layers that form the DSG sub-net to optimize it end-to-end. This sub-net consists of two layers, which takes the set-based facial representations fi's as input and outputs the reconstructed discriminative features.

The 1st layer makes the prototype prediction. Given the input deep facial representation fi, this layer outputs its indicator zi ∈ {0, 1} by dynamically projecting each input latent affinity graph to K prototypes, zi = σ(W ⊤fi), where σ is the sigmoid activation function to rectify inputs to [0, 1] and W is the DSG predictor parameter. Given the predicted zi and input fi, the 2nd layer computes the DSG loss (see Sec. 3.3) to ensure the reconstructed representation ˆ fi form reasonably compact and discriminative prototypes. ˆ fi is obtained by element-wisely multiplying the output of the 2nd layer of the DSG sub-net with the predicted multi-prototype indicator Z from its 1st layer. More details are given in Sec. 4.

3.3 Optimization

We optimize MPNet by minimizing the following two loss functions in a conjugate way.

Ranking Loss

A ranking loss is designed in MPNet to enforce the distance to shrink for genuine set pairs and increase for imposter set pairs, so that MPNet explicitly maps input patterns into the target spaces to approximate the semantic distance in the input space.

We first ℓ2-normalize the outputs from the DSG sub-net, so that all the set-based facial representations are within the same range for loss computation. Then, we use Euclidean distance to measure the fine-grained pairwise dissimilarity between ˆ fi and ˆ fj :

We further make an ensemble of the R2 distances into one energy-based matching result for each coarse-level set pair:

where β is a bandwidth parameter.

Our final ranking loss function is formulated as

Here τ is a margin, such that EG + τ < EI; EG is the distance for genuine pair; EI is the distance for imposter pair; and yp is the binary pairwise label, with 0 for genuine pair (LRanking = EG in Eq. (4)) and 1 for imposter pair (LRanking = max(0, τ -EI) in Eq. (4)).

Dense SubGraph Loss

We propose to learn dense prototypes via solving the problem defined in Eq. (1). Expanding the objective in Eq. (1) gives

Since zik, zjk ∈ {0, 1}, we have ∑Kk=1 zikaijzjk = aij only if zik = zjk = 1 for some k, i.e., the face media i and j are divided into the same prototype. Maximizing the trace in Eq. (1) is to find the partition of samples (indicated by z) to form subgraphs such that the samples associated with the same subgraph have the largest total affinity (i.e., density) ∑zik̸=0,zjk̸=0 aij. In practice, maximizing tr(Z ⊤AZ) would encourage contributions of the representations ˆ fi belonging to the same prototype to be close to each other and each resulted cluster to be far away from others, i.e., they form multiple dense subgraphs.

In Eq. (5), each element of the affinity A encodes similarity between two corresponding media, i.e.

Equivalently, the DSG learning can be achieved through the following minimization problem:

where D is the Euclidean distance matrix computed in Eq. (2).

Then we define the following DSG loss function to optimize the learned deep set-based facial representations:

Thus, minimizing the DSG loss would encourage contributions of the representations ˆ fi's belonging to the same prototype to be close to each other. If one visualizes the learned representations in the high-dimensional space, the learned representations of one face media set form several compact clusters and each cluster may be far away from others. In this way, a face media set with large variance is distributed to several clusters implicitly. Each cluster has a small variance. We also conduct experiments for illustration in Sec. 4.1.

To simplify the above optimization, we propose to relax the constraint of zij ∈ {0, 1} to 0 ≤ zij ≤ 1 by a sigmoid activation function. Thus, the DSG loss is re-defined as

We adopt the joint supervision of ranking and DSG losses to train MPNet for multi-prototype discriminative learning:

where λ is a weighting parameter among the two losses.

Clearly, MPNet is end-to-end trainable and can be optimized with BP and SGD algorithm. We summarize the learning algorithm of MPNet in supplementary materials App. B.

4 Experiments

We evaluate MPNet qualitatively and quantitatively under various settings for unconstrained set-based face recognition on IJB-A [Klare et al., 2015]. See supplementary materials App. C for full implementation details. For details of the datasets and evaluation metrics, please see App. D, E and F.

4.1 Evaluations on IJB-A Benchmark

Ablation Study and Quantitative Comparison

We first investigate different architectures and losses of MPNet to see their respective roles in unconstrained set-based face recognition. We compare 8 variants of MPNet, i.e., baseline (siamese VGGface [Parkhi et al., 2015]), w/o DSG, MPNetK∈{3,10,200,500,1000}, and MPNet (backbone: ResNet101 [He et al., 2016]).

The performance comparison in terms of TAR@FAR, FNIR@FPIR and Rank1 on IJB-A is reported in the lower panel of Tab. 1. By comparing the results from w/o DSG vs. the baseline, around 1% improvement for overall evaluation metrics can be observed. This confirms the benefits of the basic refining tricks in terms of the network structure. Compared with w/o DSG, MPNetK=500 further boosts the performance by around 3%, which speaks well for the superiority of using the auxiliary DSG loss to enhance the deep set-based facial representation learning. It simplifies unconstrained set-based face recognition, yet reserves discriminative and comprehensive information. By varying the numbers of prototypes, one can see that as K increases from 3 to 1,000, the performance on the overall metrics improves consistently when K ≤ 500. This demonstrates that the affinity-based dense subgraph learning of the proposed DSG sub-net can effectively enhance the deep feature representation capacity of unconstrained set-based face recognition. However, further increasing K does not bring further performance improvement and may even harm the performance on the overall metrics. This can be explained as follows. An appropriately large value of K will predict a sparse prototype partition indicator matrix Z, which helps reach an optimal trade-off between facial information preserving and computation cost for addressing the large variance and false matching caused by untypical faces. However, an oversize value of K will enforce the learned filters to all zero ones, which always produces invariant performance without any discriminative information. We hence set K to 500 in all experiments.

For fair comparison with other state-of-the-arts (upper panel of Tab. 1), we further replace the backbone from VGGface to ResNet-101 (bottom row) while keeping other settings the same. Our MPNet achieves the best results over 10 testing splits on both protocols. This superior performance demonstrates that MPNet is very effective for the unconstrained set-based face recognition in presence of large intraset variance. Compared with existing set-based face recognition approaches, our MPNet can effectively address the large variance challenge and offer more discriminative and flexible face representations with small computational complexity. Also, superior to the naive average or max pooling of face features, MPNet effectively preserves necessary information through the DSG learning for set-based face recognition.

Moreover, compared with exhaustive matching strategies (e.g., DCNN [Chen et al., 2016]) which have O(mn) complexity for similarity computation (m, n are media numbers of each face set to recognize) and take ∼1.2s for recognizing each probe set, our MPNet is more efficient as it operates on prototype level, which significantly reduces the computational complexity to O(K2 ), K2 << mn (K is the prototype number) and takes ∼0.5s for recognizing each probe set. Although naive average or max pooling strategies (e.g., Pooling faces [Hassner et al., 2016]) are slightly advantageous in testing time (∼0.3s for recognizing each probe set), they suffer from information loss severely. Our MPNet effectively preserves the necessary information through DSG learning for unconstrained set-based face recognition.

Qualitative Comparison

We then verify the effectiveness of our deep multi-prototype discriminative learning strategy. The predicted prototypes with relatively larger affinities within the set 1311 and set 3038 from the testing data of IJB-A split1 are visualized using t-SNE [Maaten and Hinton, 2008] in Fig. 3. We observe that MPNet explicitly learns to automatically predict the prototype memberships within each coarse-level face set reflecting different poses (e.g., the first 6 learned prototypes), expressions (e.g., the 1st and the 7th learned prototypes), illumination (e.g., the 2nd and 5th learned prototypes), and media modalities (e.g., the 5th and 6th learned prototypes). Each learned prototype contains coherent media offering collective facial representation with specific patterns. Outliers within each face set are detected by MPNet (e.g., the last learned prototypes). MPNet is learnt to enhance the compactness of the prototypes as well as their coverage of large variance for a single subject face, through which the heterogeneous attributes within each face media set are sufficiently considered and flexibly untangled. Compared with clustering-based data partition, MPNet with DSG learning is advantageous since it is end-to-end trainable, can learn more discriminative features and is robust to outliers. Learning DSG maximizes the intra-prototype media similarity and inter-prototype difference, resulting in discriminative face representations. This is significantly different from clustering (e.g., k-means) methods where only the similarity defined based on the distance to the center is considered during learning.

Finally, we visualize the verification results for IJB-A split1 to gain an insight into unconstrained set-based face recognition. Please refer to supplementary materials App. G for fully detailed visualization results in high resolution and relevant discussions.

5 Conclusion

We proposed a novel Multi-Prototype Network (MPNet) with a new Dense SubGraph (DSG) learning sub-net to address unconstrained set-based face recognition, which adaptively learns compact and discriminative multi-prototype representations. Comprehensive experiments demonstrate the superiority of MPNet over state-of-the-arts. The proposed framework can be easily extended to other generic object recognition tasks by utilizing the area-specific sets. In future, we will explore a pure MPNet architecture where all components are replaced with well designed MPNet layers, which can hierarchically exploit the multi-prototype discriminative information to solve complex computer vision problems.

Acknowledgements

The work of Jian Zhao was partially supported by China Scholarship Council (CSC) grant 201503170248. The work of Junliang Xing was partially supported by the National Science Foundation of Chian 61672519. The work of Jiashi Feng was partially supported by NUS IDS R-263-000-C67646, ECRA R-263-000-C87-133 and MOE Tier-II R-263000-D17-112.