Cascaded SR-GAN for Scale-Adaptive Low Resolution Person Re-identification

1 Introduction

Person re-identification (REID) is the task of visually matching images of the same person, extracted from nonoverlapping camera views in open surveillance spaces [Wang et al., 2016a]. Since biometric cues, such as face and gait, are usually unreliable or even infeasible in the uncontrolled surveillance environment, the appearance of individuals is Camera f mainly exploited [Ye et al., 2017; 2016]. In order to overcome the variations in illumination, occlusion and alignment, existing methods typically address the REID task by designing feature representation [Liao et al., 2015; An et al., 2015; Zhang et al., 2016; Zhao et al., 2017] or learning distance metrics [Chen et al., 2015; Wang et al., 2017b; 2017a; Yu et al., 2017].

∗ corresponding author

Figure 1: Illustration of person image resolutions with different scales in the open-space person re-identification. Three images of the same person are captured in three different camera views in the 3DPES dataset. The resolutions of these images are significantly different. The person image captured by camera b is relatively HR, but the person images captured by camera c and camera f are relatively LR. Meanwhile, the resolutions of any pair of these images are different. Intuitively, the LR gallery images should be enlarged to HR, and then the re-identification process carries on. Due to two different LRs, super-resolution (SR) modules with two upscaling factors are needed.

Generally, without regard to various low-resolutions and scale mismatching, most of these methods make an assumption that all person images have sufficiently high resolutions (HR), and they resize the images to a uniform scale before re-identification. However, it is common that the resolution of surveillance person image varies a lot, due to variations in the person-camera distance and camera deployment settings. Taking 3DPES dataset [Baltieri et al., 2011] as an example, Figure 1 shows a usual situation in open space REID, that person images are not only low resolution (LR), but also holding different scales. This gives rise to a more challenging task that given a HR probe image, the algorithm is expected to match against LR gallery images with different scales. It requires to address cross-resolution matching since LR images contain much less information with discriminative appearance details largely lost in the image acquisition process. We name this kind of problem as Scale-Adaptive Low Resolution Person Re-identification (SALR-REID).

Actually, a couple of works [Jing et al., 2015; Li et al., 2015; Jiao et al., 2018] have paid attention to LR and resolution mismatching problem. They set the probe image as a LR one, while all the gallery images are typically HR with a uniform scale. To address the mismatching of two resolutions (LR and HR), they constructed a cross-resolution relationship by a large number of training samples. However, their assumption is in a relatively ideal condition. In practice, gallery images are always not only LR, but also holding different scales. They have neglected the variation of LR scales. Multiple resolution scales require more training samples to construct relationships, and it cannot be guaranteed that those relationships work perfectly in matching. The common weakness makes these works inappropriate to address the SALRREID problem. So far, to our best knowledge, only one pioneer research [Wang et al., 2016b] investigated the SALRREID problem. Instead of recovering the missing discriminative appearance information of LR images, the method performed multiple resolution representation transformation in a pre-defined feature space. Hence, the pioneer research does not inherently solve the information mis-matching challenge, whose effectiveness still needs promotion.

Intuitively, we employ super-resolution (SR) techniques to alleviate the resolution mismatch problem. During the process of super-resolving, SR methods [Dong et al., 2016; Johnson et al., 2016; Jiang et al., 2017] supplement and recover the missing appearance information (Figure 1), so that LR gallery images and the HR probe image can be treated equally. The generative adversarial network (GAN) for image SR [Ledig et al., 2016], i.e., SR-GAN, offers an effective solution to image SR, and is also one of the most competitive SR approaches. Nevertheless, (1) a fixed SR module is not suitable for the scale-adaptive LR scenario. SR-GAN can only increase the resolution with a fixed upscaling factor, while the SALR-REID problem requires to synthesize HR images with multiple upscaling factors. (2) Generic-purpose SR methods are designed to improve image visual sense rather than the re-identification performance. The ability of SR-GAN to capture inter-image similarity is limited as it is defined based on perceptually intra-image differences. A direct connection between super resolution and re-identification may suffer from suboptimal compatibility.

In this paper, we address the SALR-REID problem by exploring re-identification and multiple cascaded SR-GANs in series with a unified framework. We name the new framework as Cascaded Super-Resolution GAN (CSR-GAN). The contributions are as follows: (1) We cascade multiple SRGANs in series, which is capable of super-resolving LR images with scale-adaptive upscaling. As far as we know, we are the first to propose a cascaded super-resolution deep network for scale-adaptive low resolutions. (2) The proposed CSR-GAN improves the integration compatibility between scale-adaptive super-resolution and re-identification, and consequently enhances the similarity of LR-HR pair during SR process. (3) For LR-HR intra-image pair, we design a common-human loss to make the super-resolved image look more like human. For LR-HR inter-image pair, we introduce the unique-human loss to make person image representation discriminative. Together with generator and discriminator losses, a joint loss function is optimized in a hybrid network architecture. Experimental results on SALR-REID datasets show the superiority of our CSR-GAN approach.

2 Revisit SR-GAN

As we know, GAN [Goodfellow et al., 2014] provides a powerful framework for generating plausible-looking natural images with high perceptual quality. SR-GAN [Ledig et al., 2016] proposes a very deep ResNet [He et al., 2016] architecture using the concept of GAN for photo-realistic image super-resolution. It sets a new state-of-the-art for image SR with a fixed upscaling factor (4×). Following [Goodfellow et al., 2014], it tries to solve the adversarial min-max problem as Eq. 1. The general idea behind this formulation is that it allows one to train a generator model GθG with the goal of fooling a differentiable discriminator DθD that is trained to distinguish super-resolved images from real images [Ledig et al., 2016].

In this formulation, ˆ I stands for the real HR image, while I stands for the LR image to be super-resolved 1 . The generator function GθG is parametrized by θG, and the discriminator function DθD is parametrized by θD. Generally, the target of previous supervised SR algorithms is commonly the minimization of the mean squared error (MSE) [Wang and Bovik, 2009] between the recovered HR image and the ground truth. Besides the MSE loss, SR-GAN also defines a perceptual loss using high-level feature maps of the VGG network [Simonyan and Zisserman, 2014], which makes the super-resolved image and HR reference image perceptually similar. However, to address the SALR-REID task, two more functionalities need to be annexed.

• To promote the ability of scale-adaptive upscaling, it requires combining multiple SR-GANs, so that scaleadaptive LR images can be enlarged to a uniform HR.
• To supplement the ability of discriminative person representation extracting, it requires plugging in the reidentification network, so that identity appearance information can be captured during SR.

1 In this paper, if image ˆ I has a hat, it means that the image is a real image. Otherwise, image I without a hat means that it is a super-resolved image.

3 Our Approach

We propose the Cascaded Super-Resolution GAN (CSRGAN) framework in a unified architecture. Figure 2 shows the architecture. Suppose that all person images have the same weight-to-height ratio. W and H respectively denote the weight and height of the original HR image. To make the architecture easy to follow, we use a three-cascaded SRGANs to enlarge the LR person image, whose resolution is lower than W2 ∗ W2 . Each generator network can enlarge the input image with a double upscaling factor. That is to say, W8 ∗ W8 , W4 ∗ W4 and W2 ∗ W2 are respectively image shapes input to three generator networks. Here, we denote I k as the input image of each generator Gk, and I k+1 = GθGk (I k ) as the output image of the generator, where k ∈ [0, 2] is the scale index of input image and the ID of sub SR-GAN. ˆ I k+1 denotes the real image, which has the same scale as the superresolved image GθGk (I k ) or I k+1 . We set rk standing for the scale ratio of LR image to HR image, then the resolution of image I k is described as rkH ∗ rkW. The scale ratios are respectively r0 = 1/8, r1 = 1/4, r2 = 1/2, and r3 = 1.

Our goal is not only to train generator functions {Gk} that estimate the corresponding HR counterpart ˆ I k+1 for a given LR input image I k , but also to train re-identification function F that extracts feature for the final HR image. Following [Ledig et al., 2016], we train the generator networks with pixel-wise loss, VGG loss, and adversarial loss, which together form the generator network loss lSR Gen . For super resolution, we also train the discriminator networks with loss lSR Dis from the perspectives of game theory, and boost the generator networks indirectly. In addition, a common-human loss lCom is designed to make person image looks better. The re-identification network is trained with unique-human loss lUni. Finally, all losses are combined together as Eq. 2.

For simplify, we balance the weights of these losses equally. The details of losses are described in the following subsections.

3.1 Generator Network Loss

The generator network loss is critical for the performance of our generator network. In general, super-resolution is commonly modeled based on the pixel-wise MSE [Dong et al., 2016]. In particular, we introduce the adversarial loss for each generator network k, and exploit the VGG loss for perceptual similarity as [Ledig et al., 2016] did. The pixel-wise loss lSRk MSE , the VGG loss lSRk VGG, and the adversarial loss lSRk Adv are demonstrated as follows.

Pixel-wise loss. The pixel-wise MSE loss is calculated as:

This is the most widely used optimization target for image super-resolution on which many state-of-the-art approaches rely [Shi et al., 2016].

VGG loss. Furthermore, a VGG loss based on the ReLU activation layers of the pre-trained 19 layer VGG network, described in [Simonyan and Zisserman, 2014], is exploited for perceptual similarity, measuring on higher semantical level which a naive MSE loss is unable to handle. It should be mentioned that the VGG losses are adopted in all the VGG networks, who share common parameters. We denote φi,j as the feature map obtained by the j-th convolution before the i-th maxpooling layer within the VGG19 network. The VGG loss is calculated as the Euclidean distance between the feature representations of a super-resolved image GθGk (I k ) and its corresponding HR image ˆ I k+1 . Here, Wi,j and Hi,j describe the dimensions of the respective feature maps with the VGG network.

Adversarial loss. Besides the losses for image content described above, we add a generative component. Along with the losses for deceiving the discriminator networks, the adversarial loss is defined based on the probabilities of the discriminator over all training samples. It is denoted as:

where DθDk (GθGk (I k )) is the probability that the superresolved image GθGk (I k ) is a real image. For better gradient behavior we minimize - log DθDk (GθGk (I k )) instead of log[1 - DθDk (GθGk (I k ))] [Goodfellow et al., 2014].

At last, we formulate the generator network loss lSR Gen as the weighted sum of afore-mentioned three kinds of losses 2 :

where α and β respectively denote the weights for the VGG loss and the adversarial loss.

3.2 Discriminator Network Loss

The discriminator network loss is defined based on the probabilities of the discriminator over all training images and super-resolved images. The discriminator should distinguish super-resolved images from real ones. Hence, the loss of Dk needs to be minimized when judging a real image ˆ I k+1 to be positive or a super-resolved image GθGk (I k ) to be negative. Then, the total loss is calculated as:

where DθDk (GθGk (I k )) is the probability that the superresolved image GθGk (I k ) is a real image, and DθDk (ˆ I k+1 ) is the probability that the original image ˆ I k+1 is a real image. For better gradient behavior we minimize - log DθDk ( ˆ I k+1 ) instead of log[1 -DθDk (ˆ I k+1 )] [Goodfellow et al., 2014].

3.3 Common-human Loss

Generally, the output of VGG network [Simonyan and Zisserman, 2014] is the category of input image. Suppose that the VGG network is able to judge each real image ˆ I 3 as a human category. To make the super-resolved I 3 look like a human, we design a common-human loss to constrain its category results to be the same as that of ˆ I 3 . The common-human loss is calculated as the Euclidean distance between category results of ˆ I 3 and I 3 . It is defined as below:

2 In this paper, if no otherwise specified, following [Ledig et al., 2016], we set α = 2 ∗ 10-6 and β = 1 ∗ 10-3 .

where c denotes the dimension index, and ψc is possibility that an image is assigned to the c-th category. The total number of categories is 1000.

3.4 Unique-human Loss

Given an original image ˆ I 3 or an super-resolved image I 3 , the output of re-identification network is z = [z1, z2, ..., zM] ∈ RM , where M is the number of person IDs. So the predicted probability of each ID label m is calculated as: p(m|I 3 ) = exp(zm) ∑Mi=1 exp(zm) . To simplify the equation, we omit the correlation between m and I 3 . The cross entropy of identification loss is formulated as below:

Let y be the ground-truth ID label, so that q(y) = 1, and q(m) = 0 for all m ̸= y. In this case, minimizing the identification loss is equivalent to maximizing the possibility of being assigned to the ground-truth class.

4 Implementation Details

The training process includes the following three steps: (1) We first initialize the re-identification network separately. We choose ResNet-50 [He et al., 2016] as the base. The ResNet50 is pre-trained with ImageNet [Russakovsky et al., 2015], and then fine-tuned with the Market-1501 [Zheng et al., 2015] dataset. (2) The cascaded generator networks are initialized with MSE losses. (3) The whole network is trained simultaneously with all the losses. For each sub SR-GAN, the settings of strides and feature maps are the same as [Ledig et al., 2016].

During the evaluation process (as Figure 3 shows), a resolution detection module is used to assign the LR gallery images to different stages of cascaded generator networks. The assignment depends on the scale ratio r of LR to HR. When a LR image is assigned to the generator network k, it will be resized to the input shape of corresponding generator network rkH ∗ rkW. After cascaded SR, the super-resolved gallery images and the HR probe image are put into the reidentification network to extract features. Finally, features are used for person re-identification.

5 Experiments

5.1 Experimental Datasets and Settings

Following [Wang et al., 2016b], the evaluation is run on two simulated person datasets SALR-VIPeR and SALR-PRID, which are based on the VIPeR dataset [Gray et al., 2007] and the PRID450S dataset [Roth et al., 2014] respectively, and the public CAVIAR dataset [Cheng et al., 2011].

SALR-VIPeR. The widely used VIPeR dataset [Gray et al., 2007] contains 1264 outdoor images obtained from two views of 632 persons. All images of individuals are normalized to a size of 128 ∗ 48 pixels. To construct the SALRVIPeR dataset, we set images from camera A as the HR probe set, whose resolution remains unchanged. While images from camera B are set as the LR gallery set, which are down-sampled randomly to different scales. The scale ratios range from 0.1 to 0.25. Some example images are shown in Figure 4(a).

SALR-PRID. The PRID450S [Roth et al., 2014] is a challenge dataset, particularly there is camera characteristics variation. It contains 450 single shot image pairs captured over two spatially disjoint camera views. All images are normalized to 168 ∗ 80 pixels. We construct SALR-PRID dataset following the way of constructing SALR-VIPeR. Some example images are shown in Figure 4(b).

CAVIAR. The CAVIAR dataset [Cheng et al., 2011] contains images of 72 individuals captured from 2 cameras in a shopping mall. This dataset is suitable for testing SALRREID, as the resolution of images captured from the second camera is much lower than that in the first camera (Figure 4(c)). Among the 72 people, 18 were only captured in a single camera view with no low resolution images, and they were thus removed. The remaining persons were used in our experiments, where a HR image of each person is selected to form the probe set, and a LR image of each person is selected to form the gallery set.

Settings. Following [Wang et al., 2016b], all datasets are randomly divided into training set and testing set. Persons for training and testing are respectively 532 and 100 (SALR-VIPeR), 400 and 50 (SALR-PRID), and 44 and 10 (CAVIAR). The probe set consists of all HR images per person. LR images are randomly downsampled and selected to construct the gallery set. Cumulative Matching Characteristic (CMC) curves [Wang et al., 2007] were used to calculate the average performance, and the value of CMC@k indicates the percentage of the real match ranked in the top k.

5.2 Evaluation on Scale-Adaptive SR

In this subsection, we prove that the proposed method is applicable for scale-adaptive SR. In Figure 5, we show some subjective results on scale-adaptive LR images. The SR results are generated respectively by bicubic, CSR-GAN without common-human loss, and CSR-GAN with commonhuman loss. Here, we choose images from SALR-VIPeR and SALR-PRID as examples. From the figure, we can find that CSR-GAN is effective for scale-adaptive SR, and the effectiveness varies with different LR scales. We can also find that the generated result by CSR-GAN with common-human loss looks better than without common-human loss. For example, for image 2, the SR image without common-human loss drops out its human shape a little. For the SR images 7, 10, 11 and 12, there are less noises in the image background with common-human loss, where we guess that CSR-GAN pays more attention to the human body and additional information for the background are smoothed. Hence, the commonhuman loss is every useful for person image SR.

Meanwhile, we performed a Mean Opinion Score (MOS) test to quantify the ability of scale-adaptive super resolution.

Specifically, we asked 5 raters to assign an integral score from 1 (bad quality) to 5 (excellent quality) to the superresolved images. Each rater rated all the testing images of three datasets. We found no significant differences between the ratings of the identical images. As images with different resolution scales will go to different process of CSR-GAN, all the images are divided into different groups depending on their scale ratio. The experimental results of the conducted MOS tests are summarized in Table 1. As can be seen, the proposed method outperforms the general SR methods (nearest and bicubic) in all the scale ratio groups. It should be mentioned that we did not compared with the other supervised learning SR methods, because most of those methods just enlarge the image with a fixed upscaling factor.

5.3 Evaluation on SALR-REIDDatasets

In this subsection, we prove that the proposed method is suitable for the SALR-REID problem. We evaluated the effectiveness of the proposed method by comparing with one of the most popular person re-identification method LOMO [Liao et al., 2015], a basic person image descriptor method with RGB color, and the state-of-the-art method SDF [Wang et al., 2016b], on the SALR-VIPeR, SALR-PRID and the CAVIAR datasets, respectively. In particular, when we used LOMO and Color to extract features, all the scale-adaptive LR gallery images were resized to the uniform HR with bicubic. The obtained results are shown in Figure 6. As can be seen, the general person re-identification method LOMO is almost no effect, even performs worse than the basic color descriptor. Consequently, general descriptors are not suitable for the SALR-REID problem. Meanwhile, we can also find that our approach has improvements on all the three datasets, compared with LOMO, Color and SDF. The promotions are significant on the SALR-VIPeR and SALR-PRID dataset, where the scale-adaptive LR problem is serious.

5.4 Comparison with State-of-the-art LRMethods

Recently, some approaches have paid attention to LR problem, and proposed algorithms to address the resolution mismatching problem, such as JUDEA [Li et al., 2015], SLD2L [Jing et al., 2015] and SING [Jiao et al., 2018]. These methods assume that all gallery images are HR, and probe images are LR with a fixed down-sampling factor. For example, MLR-VIPeR [Jiao et al., 2018] was constructed by down-sampling images with a ratio picked from { 1 2 , 13 , 1 4 }. To evaluate the ability of addressing resolution mismatching, we compared our method with these methods on the MLRVIPeR dataset. Table 2 shows the results. From the table, we can see that although our method is proposed for SALRREID problem, it outperforms related state-of-the-art methods on fixed the resolution mismatching problem with a big margin.

6 Conclusion

This paper focuses on a new issue, i.e., scale-adaptive low resolution person re-identification. To promote the ability of scale-adaptive upscaling and image feature extracting, we propose a new framework CSR-GAN. Besides generator network and discriminator network losses, a common-human loss and a unique-human loss are introduced, and optimized in a hybrid network architecture. Using extensive experiments, we have confirmed that CSR-GAN is capable of superresolving person images with adaptive upscaling, achieves a considerable promotion on SALR-REID datasets, and outperforms state-of-the-art LR person re-identification methods.

Acknowledgments

This work was supported by JST CREST Grant Number JPMJCR1686, Japan.