CoSegNet: Image Co-segmentation using a Conditional Siamese Convolutional Network

1 Introduction

Image co-segmentation plays a significant role in computer vision since it identifies images that have common objects and jointly segments out those objects from them (Fig. 1). Further, co-segmentation has the potential to improve result of single image segmentation as it integrates information from a group of similar images [Rother et al., 2006]. It can be used as a tool for image retrieval, annotation, object recognition and person re-identification.

Existing common challenges in the co-segmentation problem are to find suitable features when the appearance, shape and pose of foregrounds vary significantly, foreground has notable similarities with background, and images contain missing foregrounds. There has been a significant amount of work on unsupervised co-segmentation [Hati et al., 2018; Joulin et al., 2012; Vicente et al., 2010]. But, in general, these methods are sensitive to the selected hand-crafted features and require exhaustive feature tuning. There also have been some semi-supervised approaches [Batra et al., 2010; Wang and Shen, 2016] where users provide sparse foreground and background labels on training images as scribbles and the co-segmentation algorithm learns background and foreground appearance models from these images. To handle the aforementioned challenges, these methods have to iterate between learning appearance models and acquiring human inputs, which is difficult.

∗We acknowledge the support provided by Bharti Centre for Communications, Department of Electrical Engineering, IIT Bombay. Sayan Banerjee is the corresponding author.

In order to solve those difficulties, we propose a CNN based simple end-to-end architecture for co-segmentation (Fig. 2). Our model utilizes a siamese convolutiondeconvolution (encoder-decoder) network, which takes a pair of images as input, produces intermediate convolutional features using feature encoders and conditionally maps those features into corresponding co-segmentation masks using decoders. These masks are used to extract the common object (Fig. 1). We use a siamese metric learning network that learns an optimal latent feature space where objects belonging to the same class are closer and objects from different classes are well separated, without using semantic class labels. Furthermore, we use a decision network on top of the metric learning network to produce a binary label, indicating the presence or absence of a common object in the input image pair. Together these two networks condition the siamese encoder-decoder network to perform an accurate co-segmentation depending on the presence of a common object in the input image pair.

During training, the binary ground truth masks of the training images guide the encoder-decoder network to differentiate common objects from the background based on the learned features. The metric learning part guides the encoderdecoder network to reduce intra-class object distance and increase inter-class object distance. As identifying outlier images is very difficult during co-segmentation, we propose a novel training strategy. In the case of positive samples (an image pair with a common object), we train the whole network. However for negative samples (an image pair with no common object), we only train the complete metric learning, decision, encoder and certain part of decoder network (please see Sec. 3.5 for more detail). During testing, the decision network predicts the presence or absence of a common object in them and accordingly the siamese encoder-decoder network estimates the corresponding co-segmentation masks.

The main contributions of this paper are:

• We propose a twofold siamese network architecture that is class agnostic.
• One of them being a novel conditional siamese encoderdecoder network and the other being a siamese metric learning network followed by a decision network, which takes input from intermediate layers of decoder network, as opposed to conventional approaches that use the encoder output as input. This design helps to generate better co-segmentation masks through image similarity.
• We propose a novel training strategy which helps the network to discard outlier images and boost performance with low amount of training data.

We perform extensive experiments and ablation studies on various challenging co-segmentation datasets such as the PASCAL-VOC dataset [Everingham et al., 2010], the Internet dataset [Rubinstein et al., 2013], the MSRC dataset [Rubinstein et al., 2013], and demonstrate significant improvement in performance over state-of-the-art methods.

2 RelatedWork

The unsupervised object co-segmentation problem was introduced by [Rother et al., 2006] where they provide a solution for handling two images with a Markov random field (MRF) based generative. [Joulin et al., 2010] solved cosegmentation using clustering by finding discriminative object features, whereas, the method in [Vicente et al., 2010] uses Boykov-Jolly model and a dual decomposition technique based optimization. The method in [Wang et al., 2013] represents each image in a linear functional space and learns a linear transformation between different functional space to perform co-segmentation. Many researchers have also been trying to solve the co-segmentation problem with different degrees of supervision. [Rubio et al., 2012] trained SVM classifier on top of a Gaussian Mixture Model to find out correspondence between different regions of input images. The method in [Rubinstein et al., 2013] uses dense correspondences and saliency. [Hsu et al., 2018] also used saliency along with a pretrained convolution network in an unsupervised manner to solve co-segmentation. [Chen et al., 2014] used segmentation prior (seed) with learned detector. The method in [Quan et al., 2016] uses CNN features obtained from VGG

net and handcrafted features for superpixels and subsequently implements two separate graph-cuts for background and foreground superpixels. The methods in [Batra et al., 2011; Batra et al., 2010; Dong et al., 2015; Wang and Shen, 2016] propose semi-supervised solutions using sparse scribbles. [Yuan et al., 2017] proposed a deep neural network based end-to-end co-segmentation model where they extracted a set of object proposals and applied deep conditional random field to find co-occurring objects. Subsequently, they segmented objects from these proposals independently. Therefore, their method is not robust against outliers. A very recent CNN based deep co-segmentation network presented in [Chen et al., 2018] additionally used attention for solving the problem. The work in [Li et al., 2018] also uses a siamese CNN to estimate co-segmentation masks. Different from their architecture, our proposed siamese encoder-decoder network is conditioned by a siamese metric learning network and decoder network that learn similarity across common objects (discussed in detail in the next section) which helps us in obtaining significantly better co-segmentation performance.

3 Proposed Network Architecture

Given a pair of input images, our aim is to segment the common objects from them. It should be noted that our problem setting is quite challenging, since we assume that the images do not always contain common objects, and when there is a common object, their poses and appearances may vary significantly. Humans have the ability to identify objects of the same class even if they significantly differ in pose, shape and appearance. To capture this aspect, we incorporate a metric learning approach to learn a latent feature space, which ensures that objects from the same class will be projected very close to each other and that of different classes will be projected far apart at least by a margin irrespective of their shape, pose and appearance variations. Furthermore, the object should be distinguishable from its background. Let {Ii}2i=1 ∈ Rq×q be a pair of input images and {Mi}2i=1 ∈ Rq×q be the corresponding ground truth binary masks highlighting the regions of common object. Our objective in the proposed network is to estimate such masks { ˆMi}2i=1 using a conditional siamese encoder-decoder network. The proposed network architecture is shown in Fig. 2(a). It is composed of three major components: a conditional siamese encoderdecoder network, a fully-connected siamese metric learning network and a decision network.

3.1 Conditional Siamese Encoder-Decoder Network

The proposed encoder-decoder network has a convolution feature encoder and co-segmentation mask decoder. The siamese encoder consists of two identical feature extraction CNNs with shared parameters and is built upon the VGG 16 architecture. The input image pair I1 and I2 (224 × 224) is passed through the encoder network, which is composed of 13 convolutional layers (conv) and five max-pooling (MP) layers as shown in Fig. 2. The output of each encoders is a high level semantic feature map f1 or f2, having 512 channels with a spatial size of 7 × 7.

The siamese decoder block that follows the encoder, performs the task of producing foreground masks of the common objects. It consists of two identical deconvolution networks. The input to this network is the semantic feature map pair f1 and f2 produced by the encoder. The decoder network is formed by five spatial interpolation layers with 13 transposed convolution layers (convT) as shown in Fig. 2 [Bottom]. In the encoder, we have used max-pooling as it makes the extracted features spatially invariant and contextual. Doing so, we loose the spatial resolution of the input images. The role of the decoder network is to transform these low resolution feature maps to co-segmentation masks with resolution equal to that of the input. To increase the size of the feature maps, we use nearest neighbour interpolation (NNI), which is fast compared to bi-linear or bi-cubic interpolation. However, this way of increasing size introduces blurring and spatial artifacts. Therefore after each NNI, transposed convolution has been performed to reduce these artifacts. All deconvolution or transposed convolution layers except the final layer are followed by a ReLU operation. The final deconvolution layer produces two single channel maps with size 224 × 224, which are converted to co-segmentation masks ˆM1, ˆM2 by sigmoid function. The output layer of this network (see Fig. 2) is gated by the binary output of the decision network (discussed in Sec. 3.3) to produce a conditional siamese convolutional network. The conditional parameter is used during the feed-forward and backpropagation stages, as detailed in Sec. 3.5.

3.2 Siamese Metric Learning Network

It consists of two fully connected layers with dimensions 128 and 64, respectively. The first layer has ReLU as non-linearity and the second layer does not have any non-linearity. This network takes input fu1 and fu2 from the siamese decoder network and outputs a pair of feature vectors fs1 and fs2 that represent the objects in the learned latent space. Since the deconvolution layers at the middle of the decoder network capture sufficiently enough object information, we use the output of the ninth deconvolution layer (56 × 56 and having 256 channels) as the input to this network. We use global average pooling (GAP) over each channel of the deconvolution layer to get two 256 dimensional vectors. To train the network, we use standard triplet loss Sec. 3.4. During backpropagation this also updates the nine decoder deconvolution layers and thirteen encoder convolution layers as shown in Fig. 2 which leads to better masks.

3.3 Decision Network

During testing, we are required to infer whether the input image pair contains common objects. We achieve this using a decision network that detects the presence or absence of common objects. This network is built upon the feature similarity network and has two fully connected layers with dimensions 32 and 1, respectively. The input to the network is a 128 dimensional vector obtained by concatenating fs1 and fs2. The first layer is associated with a ReLU non-linearity. The second layer is associated with a Sigmoid function that gives a probability signifying presence or absence of common object in the two input images. During testing, we threshold the probability and convert it to a binary label. For an input image pair if the decision network predicts a binary label one, we compute the corresponding binary masks at the output of the siamese decoder network.

3.4 Loss Function

The loss function used to train the proposed network is

where L1, L2 and L3 are the losses used for training the siamese encoder-decoder network, the siamese metric learning network and the decision network, respectively.

Given a set of positive and negative pair of images {(I a i , I p i ), (I a i , I n i )} (where I a i is an anchor), corresponding pair of ground truth masks {(M a i ,M p i ), (M a i ,M n i )} and the predicted masks {( ˆM a i , ˆM p i ), ( ˆM a i , ˆM n i )} obtained from the sigmoid layer of the decoder, we use pixel-wise binary cross entropy loss to train the encoder-decoder network as follows

where ˆM(j, k) and ˆM(j, k) is the value of the (j, k)-th pixel of the predicted and true mask, respectively. The loss of the metric learning network is the standard triplet loss given as

α is a scalar valued margin. To train the decision network, we use binary cross entropy loss given as

yr = 1 and 0 for a positive (I a i , I p i ) and negative pair (I a i , I n i ), respectively. And ˆ yr is the predicted label obtained from it's final sigmoid layer.

3.5 Training Strategy

As mentioned earlier, at the time of training we backpropagate loss L1 only for positive samples, and losses L2 and L3 for both positive and negative samples as shown in Fig. 2. An image belonging to a positive sample should produce an object mask. If the same image is part of a negative sample (outlier images), it is required to produce a null mask. Thus the decoder network is forced to produce two different masks for one image at different instances. For negative examples it is not required to produce any mask at all since the decision network notifies the presence of a common object. Hence, the role of the decision network is to reduce the overall difficulty level of the deconvolution layers by making them to produce object masks only for positive samples. It helps to train the network and also improves the performance as shown in Fig. 6. To summarize, we train the entire network for positive samples (yr = 1) and a part of the network for negative samples (yr = 0), thus making the mask estimation of the siamese network a conditioned one.

During testing, we obtain the co-segmentation masks by multiplying the output of the decoder network with the predcited label (ˆ yr) of the decision network because of the conditional network. Ideally, the output of the decision network for any positive example is one and that for any negative example is zero. Hence, for positive samples we obtain common object masks and for negative samples we obtain null masks, as desired. We show our experimental results in Fig. 6.

4 Experimental Results

We evaluate co-segmentation performance, using two common metrics. The first one is Precision, which is the percentage of correctly segmented pixels of both the foreground and the background. The second one is Jaccard Index, which is the intersection over union of the co-segmentation result and the ground truth common foreground segment. To evaluate the performance of the proposed co-segmentation algorithm on a set of k input images I1, I2, ..., Ik, we perform co-segmentation on all such pairs and then report the average Precision and Jaccard index computed over all such pairs as the final co-segmentation accuracy for the given set. We evaluate the proposed method on three challenging cosegmentation datasets Pascal-VOC, Internet and MSRC and compare with state-of-the-art methods.

4.1 Implementation Details

We initialize our network with weights trained on Imagenet dataset. We use stochastic gradient descent as our optimizer and fix the learning rate and momentum at 0.00001 and 0.9, respectively for all three datasets. For Pascal-VOC and MSRC datasets, we set the weight decay to 0.0004 and for Internet we set it to 0.0005. At the time of training, we follow [Schroff et al., 2015] for generating samples. For the case of positive samples, we set the weights w1 = w2 = w3 = 0.33 in (1) and for negative samples we set w2 = w3 = 0.5

and w1 = 0 since L1 is not backpropagated, as explained in Sec. 3.5. Due to memory constraint, we use a batch size of 3. Note that each input sample of the batch is a pair of input images, either positive or negative. We resize all the input images to 224 × 224 and set the margin α to 1.

4.2 PASCAL-VOCDataset

The PASCAL-VOC dataset [Everingham et al., 2010] has 20 classes and is one of the most challenging datasets due to the significant intra-class variations and presence of background clutter. We randomly split the dataset in the ratio of 3:1:1 for training, validation and testing sets. Since there is no standard split available, we repeat this splitting process 100 times and report the average performance computed over 100 such different test sets.

Analysis. In Table 1, we show comparative results of the proposed method with state-of-the-art methods and the proposed method significantly outperforms existing methods. The performance is improved by at least eight percent in terms of Jaccard Index and four percent in terms of precision. This can be explained by the fact that our model involves convolutiondeconvolution with pooling operation, which involves a high degree of context for feature computation. Furthermore, the ground-truth mask helps to localize the foreground objects and with that the metric learning network learns a latent feature space where common objects come closer irrespective of their pose and appearance variations. Hence, the method becomes robust to pose and appearance changes.

4.3 Internet Dataset

The Internet dataset [Rubinstein et al., 2013] has three classes: Airplane, Car and Horse. Each class also contains some outlier images. We use a subset of 100 images per class for our experiment as it is a common setting [Quan et al., 2016; Li et al., 2018; Hsu et al., 2018; Yuan et al., 2017].

Analysis. Comparative results of the proposed method with

other state-of-the-art methods are shown in Table 2. It can be seen that the proposed method outperforms other deep learning based methods and unsupervised methods. The method in [Yuan et al., 2017] is heavily dependent on extracted object proposals. Furthermore, after obtaining proposals that contain common objects, the segmentation of those proposals are done independently. Moreover, as observed in [Yuan et al., 2017] itself the dense-CRF based model may worsen the cosegmentation result. Our method has some similarities with the method in [Li et al., 2018], but our use of metric learning with a decision network as opposed to a mutual correlator as proposed in their work makes our model faster by 6 times per epoch, and this along with conditional siamese encoderdecoder increases co-segmentation performance by at least 6 to 8 percent from [Li et al., 2018]. We show visual results of the proposed method in Fig. 4.

4.4 MSRCDataset

We evaluate our method on a subset of the MSRC dataset, which has been widely used by previous methods to evaluate co-segmentation performance. We select the same classes, which are cow, plane, car, sheep, bird, cat and dog, as reported by [Vicente et al., 2010; Rubinstein et al., 2013; Wang et al., 2013]. Each class has 10 images and there is a single object in each image. The objects in each class have color, pose and scale differences. The experimental protocol

and parameters are same as that of the Pascal-VOC dataset. Analysis. Comparative results of the proposed method with other state-of-the-art methods are shown in Table 4. The model was trained on the Pascal-VOC dataset since the MSRC dataset does not have sufficient number of samples for training. Yet the proposed method outperforms other methods by at least five percent in terms of Jaccard index. In Fig. 5,

we show visual results of the proposed method.

4.5 Ablation Study

To show the role of our fully connected siamese metric learning network and decision network, we create a baseline model CoSegNet-base by removing the metric learning and decision networks from the proposed architecture. We concatenate features f1 and f2 along their channels to make a feature map with 1024 channels for image I1 and feed it to the corresponding decoder network. The same is done for image I2. It should be noted that, for our baseline model, we train the whole siamese encoder-decoder network for negative samples also using null mask. In Table 5, we compare the baseline model with our proposed model on different cosegmentation datasets. It can be seen from Table 5 that the proposed conditional siamese encoder-decoder network performs significantly better which justifies the inclusion of the fully connected siamese metric learning network and decision network with a novel training strategy. The advantage of the proposed CoSegNet architecture over the CoSegNet-base architecture is visually illustrated in Fig. 6. Different class objects in the image pair are incorrectly detected as common objects by CoSegNet-base, whereas CoSegNet correctly detects that there is no common object in the image pair. In Table 6, we compare the performance of our proposed model by feeding input features, obtained from different layers of the siamese deconvolution network, to the siamese metric learning network. In Table 6, f1 and f2 are the output feature maps of the siamese encoder network and cT3, cT6, cT9 and cT11 are the output feature maps of the third, sixth, ninth and eleventh deconvolution layers, respectively. The model performs the best for cT9 because (a) sufficient object information has been fed to the input of the metric learning module, (b) the number of deconvolutional layers (cT11 and cT13) dedicated for producing co-segmentation masks is optimal.

5 Conclusion

In this paper, we present a novel and efficient CNN-based architecture for solving image co-segmentation. Based on a conditional siamese encoder-decoder architecture, combined with a siamese metric learning and a decision network, we achieve better than state-of-the-art performances on various datasets, and demonstrate good generalization performance on segmenting objects of the same classes across different datasets, and robustness to outlier images.