Rethinking Loss Design for Large-scale 3D Shape Retrieval

1 Introduction

3D shape retrieval is a fundamental problem in 3D shape analysis communities, spanning wide applications from medical imaging to robot navigation. With the development of Convolutional Neural Network (CNN) and the emergence of large-scale 3D repositories [Chang et al., 2015; Wu et al., 2015], numerous approaches are proposed [Johns et al., 2016; Su et al., 2015; Xu et al., 2018; Li et al., 2019] which significantly boost the performance in 3D shape retrieval task. Among these methods, view-based methods have achieved the best performance so far. In view-based 3D shape retrieval, images are first projected from different viewpoints of a 3D shape, and then they are passed into CNNs to obtain the discriminative and informative shape representation. The crucial issue of 3D shape retrieval is how to obtain ideal shape representations: discriminative among different categories and clustered within the same class.

∗They contributed equally to this work

†Corresponding author

The majority of view-based methods like MVCNN [Su et al., 2015] train a CNN to extract shape representations with the standard softmax loss. While softmax loss does not explicitly enforce the distance between features in the embedding. To obtain ideal shape representations, [He et al., 2018] recently introduce deep metric loss functions to the 3D shape retrieval task. Such metric learning techniques, like center loss [Wen et al., 2016], triplet loss [Schroff et al., 2015] and triplet-center loss (TCL) [He et al., 2018], are conducted in the Euclidean space and could remarkably boost the retrieval performance. However, these loss functions are designed by Euclidean margin, which is difficult to be determined because of the large span of Euclidean distance and has weak geometric constraint due to the variance of feature vector magnitude. With more clear geometric interpretation, metric loss functions based on cosine margin like coco loss [Liu et al., 2017]

are particularly popular in the image retrieval and face verification communities. Considering the superior geometric constraint of cosine margin, we also introduce the coco loss to 3D shape retrieval in our experiment. The drawbacks with these methods lie in the complicated back-propagation calculation and the unstable parameter update, especially for large-scale 3D shape datasets. Therefore, we seek for a simple and stable metric loss with clear geometric interpretation to improve the discriminability of shape representations.

In this work, our intuition is that, for an ideal embedding space, the features of the same class should be in a line while at least orthogonal to other features. Meanwhile, the simplicity and stability of the loss function must be taken into consideration in the loss design for large-scale dataset. To this end, we propose a new Collaborative Inner Product Loss function (CIP Loss) to jointly optimize intra-class similarity and inter-class margin of the learned shape representations. In the loss design, we directly adopt the simple inner product which could provide distinct geometric constraints (see Fig. 1) and enforce learned visual features f to satisfy two conditions:

where y is the label and Dot(·, ·) means the inner product between two vectors. On one hand, CIP loss encourages visual features of the same class to cluster in a linear subspace where the inner product between features tends to infinity, indicating the maximum extent of linear correlation among features. On the other hand, inter-class subspaces are requested to be at least orthogonal, meaning that for each category subspace, other category subspaces are squeezed to another half-space of its orthogonal hyperplane. In particular, if there are six categories in three-dimensional feature space, CIP loss would make all categories divide the whole embedding space equally as shown in Fig. 1. Specially, in order to save computation, the inner product between visual features is replaced by the inner product between visual features and category centerlines in our method.

We show in Fig. 2 the effect of different loss functions on the feature distribution. Compared with previous metric learning techniques in 3D shape retrieval community, the proposed method has the following key advantages. 1) CIP loss could provide more explicit geometric interpretation for the embedding than approaches based on Euclidean distance, see Fig. 2(b)(c)(e). 2) Compared with popular cosine margin, CIP loss is simple and elegant to implement without margin design and normalization operation, which can stabilize the optimization process. 3) CIP loss is flexible to be plugged into existing off-the-shelf architectures, where it can work standalone or in combination with other loss functions.

In summary, our main contributions are as follows.

• We propose a novel metric loss function, namely collaborative inner product loss (CIP Loss), which adopts elegant inner product between features to performmore explicit geometric constraints on the shape embedding.
• Two components of CIP loss are proposed, namely Cluster loss and Ortho loss. Cluster loss guarantees visual

features of the same class to cluster in a linear subspace, while Ortho loss enforces inter-class subspaces are at least orthogonal, making shape representations more discriminative.

• Our method achieves the state-of-the-art in the largescale datasets, ModelNet and ShapeNetCore55, showing the effectiveness of CIP loss.

2 RelatedWork

In this section, we briefly review some view-based approaches of 3D shape retrieval and introduce some typical deep metric learning methods in this domain.

2.1 Multi-view 3D Shape Retrieval

3D shape retrieval methods could be roughly divided into two categories: model-based methods and view-based methods. Although the model-based methods [Xie et al., 2017; Qi et al., 2017; Shen et al., 2018] can capture the 3D structure of original shape by various data form, their performances are typically lower than view-based methods due to the complexity of computation and the noise in shape representation.

The view-based 3D shape retrieval methods use rendered images to represent the original shape. MVCNN [Su et al., 2015] is a typical method adopting CNN to aggregate rendered images. In this method, the rendered images from different views are pooled to generate the shape feature. [Huang et al., 2018] adopts a local MVCNN shape descriptors, which use a local shape descriptor of each point on the shape to analyze the shape. Recently proposed GVCNN [Feng et al., 2018] separates views in groups by intrinsic hierarchical correlation and discriminability among them, which largely improves the performance on the 3D shape retrieval. Another technique used in 3D shape retrieval is re-ranking. GIFT [Bai et al., 2016] adopts CNN with GPU acceleration to extract each single view feature and proposes the inverted file to reduce computation for fast retrieval.

The aforementioned methods are trained under the supervision of softmax loss which is not always consistent with retrieval task. [He et al., 2018] recently puts forward TCL that combines the center loss and triplet loss, which achieve state-of-the-art results on various datasets. This shows that deep metric learning plays an important role in the 3D retrieval task.

2.2 DeepMetric Learning

As a key part of deep learning framework, loss design has been studied widely in retrieval and other domains. Most commonly used loss functions in 3D shape retrieval are designed in Euclidean space, such as triplet loss, center loss and TCL. Triplet loss [Schroff et al., 2015] forces inter-class distance exceed intra-class distance by a positive Euclidean margin which is widely applied in face recognition. However, the original triplet loss has the problem in computation cost that the number of triplets grows cubically with the dataset. To tackle this problem, many improved version [Song et al., 2016; 2016] are proposed in various domains. [Song et al., 2016] is an efficient approach that proposes an algorithm for taking full advantage of the training samples by using the matrix of pairwise distances and achieves high performance on image retrieval. Another popular metric loss is center loss [Wen et al., 2016] that tries to gather the same class features in one point which is called center of the class. Combining the advantages of triplet loss and center loss, TCL [He et al., 2018] is proposed for 3D shape retrieval which gains a better performance than the other loss functions. However, they adopt the Euclidean margin which is difficult to design and has a weak geometric constraint.

On the other hand, cosine margin is recently proposed which are popular in face recognition. As a typical method, coco loss [Liu et al., 2017] advocates that the weakness of softmax loss is in the final classification layer. By weight normalization and feature normalization, it introduces cosine margin to reduce intra-class variation. Although the cosine margin has more clear geometric interpretation than Euclidean margin, it also brings instability in the optimization.

3 Methods

Deep metric learning aims to train a neural network, denoted by fθ(·), which maps the original data onto a new embedding space F ∈ Rn . For writing convenience, we use fi ∈ Rn to represent fθ(xi) which is the extracted feature from CNN. In this work, our goal is to make F ideal, which means that the intra-class angular distance is 0 while the inter-class angular distance is at least π2 . To fulfill this goal, the loss design needs to consider two necessary tasks: (1) Enlarging the inter-class distances between features and (2) Reducing the intra-class distances between features. Then, the CIP loss could be formalized as the sum of pull term and push term corresponding to Cluster loss and Ortho loss respectively:

where λ is a trade-off hyperparameter.

To achieve the aforementioned goal, a natural idea is adopting cosine distance which needs normalization of feature vectors, while this operation would bring complexity in backpropagation and instability in optimization that we will discuss in Sec. 3.3. In this article, we use the inner product of two vectors as the pairwise similarity for the sake of simplicity and stability:

3.1 Proposed Loss Functions

Given a training dataset X ∈ RD , let Y = {y1, y2, ..., y|X|} denote the corresponding label set, yi ∈ {1, 2, ...,K} where K is the number of classes.

The Cluster loss constrains the same class features in a linear subspace, i.e. a line in F. Inspired by center loss, we define this line, denoted by ck ∈ Rn , as the centerline of the kth class. Different from the interpretation of "center" in center loss, the centerline here represents the direction of the corresponding class features. Given a batch of training data with M samples, the pull term named Cluster loss is defined as:

where d = 2 is a constant value for numerical stability. Cluster loss encourages fi to pull its corresponding centerline cyi . And in turn, it forces the centerline to concentrate its corresponding features.

The design of Ortho loss considers that the different class features are at least orthogonal, according to condition 2. As centerline represents the direction of the corresponding class, we penalize the features which are not orthogonal to other negative centerlines. The push term named Ortho loss is formulated as:

Ortho loss pushes fi to be at least orthogonal to all other centers so that the inter-class distance will increase. The generated feature distribution of our method is shown in Fig. 2(e).

3.2 Backpropagation

Gradient of Feature Vector

Owing to the simple formulation of loss functions and adoption of the inner product, the formulas of gradient for feature are also simple. For writing convenience, we denote (·)+ = max(·, 0). We use δ(cdt) = 1 to indicate cdt is true and δ(cdt) = 0 otherwise. The gradient of fi for Lortho is foumulated as follows:

As for Lcluster, the existence of d > 0 is to adjust the gradient of function 1x which explodes on x = 0. We set d = 2 in our implementation. However the gradient function still has this problem when f T i · cyi is near to -d. In fact, the original formulation is:

Eq. 8 becomes ∞when f T i · cyi tends to be -d. Thus we clip the term f T i · cyi by 0 so that the optimization becomes stable. Concretely, we use a surrogate gradient in backpropagation:

.

Gradient of Centerline

The gradient form is similar to that of feature vector since the inner product is a symmetric function. In the same way, we adopt a surrogate gradient to evade gradient explosion for Lcluster .

For Lortho, we replace the gradient of Lortho with respect to ci by a "average" version which can further stablize the update of center:

3.3 Discussion

The loss function is essential for guiding the optimization that has a direct influence on the feature distribution. And in the training process, the convergence and the stability should be guaranteed, which poses challenges to the loss design. In this section, we will illustrate the motivation and the procedure of loss design in detail.

Inner Product without Normalization

In a typical CNN, the similarity between a deep feature vector fi and a layer weight wj is encoded by the inner product: s = f T i ·wj = ||fi||2||wj||2·cos(fi,wj). In order to eliminate the effect of norms on similarity, some methods adopt normalization operations. However, weight normalization leads to instability. We can infer from the gradient of weight after normalization:

From Eq. 12 we can see that small value of w will lead to gradient explosion. In comparision, for the inner product, the gradient is ∂(wT ·f) ∂w = f which is more stable and computationally efficient. So we choose the inner product for the sake of stabilization in the training process. As for leaving out feature normalization, we take the same consideration.

In addition, this form doesn't need margin design or, from another perspective, we can regard the angular margin as 0 and π2 in two terms corresponding that the inner product is∞ and 0.

Combination with Other Loss Functions

Both of Cluster loss and Ortho loss can combine solely with softmax loss and bring improvement on performance. It is worth pointing out that Ortho loss cannot maintain the centerlines solely since it focuses on enlarging inter-class distance. When we combine Lortho with other loss functions, the centerlines will diverge. In this case, we employ the batch version:

This batch version directly conducts the optimization on feature distribution without centerlines. The backpropagation formula of batch version is similar to Eq. 7 that we only need to replace the role of ck by fj. We will discuss the performance difference of these two versions in the Sec. 4.2.

4 Experiment

In this section, we evaluate the performance of our method on two representative large-scale 3D shape datasets. We first compare the results with the state-of-art methods. We also provide massive experiments to discuss the comparison and the combination with other metric loss functions. In the last part, we investigate the influence of the hyper-parameters and visualize the experiment results by dimension reduction.

4.1 Retrieval on Large-scale 3DDatasets

Implementation detail. The experiments are conducted on Nvidia GTX1080Ti GPU and our methods are implemented by Caffe. For the structure of the CNN, we use VGG-M [Chatfield et al., 2014] pre-trained on ImageNet [Deng et al., 2009] as the base network in all our experiments. We use the stochastic gradient descent (SGD) algorithm with momentum 2e-4 to optimize the loss and the batch size is 100. The initial learning rate is 0.01 and is divided by 5 at the 20th epoch. The total training epoch is 30. The centerlines are initialized by a Gaussian distribution of mean value 0 and standard deviation 0.01. Firstly, we render 36 views of each model by Phong reflection to generate depth images which compose our image dataset. The size of each image is 224x224 pixels in our experiment. Then, we randomly select images from the image dataset to train our CNN. In the test phase, the features of the penultimate layer, i.e. fc7, are extracted for the evaluation. We obtain the shape feature vector by averaging all view features of this shape. The cosine distance is adopted as the evaluation metric.

Dataset. We select two representative datasets, ModelNet and ShapeNetCore55, to conduct the evaluation of our methods. 1) ModelNet Dataset: this dataset is a large-scale 3D CAD model dataset composed of 127,915 3D CAD models from 662 categories. ModelNet40 and ModelNet10 are two subsets which contain 40 categories and 10 categories respectively. In our experiment, we follow the training and testing split as mentioned in [Wu et al., 2015]. 2) ShapeNetCore55: this dataset from SHape REtrieval Contest 2016 [Savva et al., 2016] is composed of 51,190 3D shapes from 55 common categories divided into 204 sub-categories. We follow the official training and testing split to conduct our experiment. The evaluation metrics used in this paper include mean average precision (MAP), area under curve (AUC), F-measure (F1)

and normalized discounted cumulative gain (NDCG). Refer to [Wu et al., 2015; Savva et al., 2016] for their detailed definitions.

Comparison with state-of-the-arts. On ModelNet40 dataset, we choose 3D ShapeNets [Wu et al., 2015], DeepPano [Shi et al., 2015], MVCNN [Su et al., 2015], PANORAMA-NN [Sfikas et al., 2017], RED [Song et al., 2017], Siamese CNN-BiLSTM [Dai et al., 2018], GVCNN [Feng et al., 2018], TCL [He et al., 2018] and GIFT [Bai et al., 2016] methods for comparison. The experimental results and comparison among different methods are presented in Tab. 1. Our method (LCIP + center loss) achieves retrieval AUC of 88.21% and MAP of 87.22% which is the best among different methods. Besides, as the current state-of-the-art view-based method on ModelNet40, TCL is trained on VGG 11 which has 3 more convolution layers than VGG-M and adopts batch normalization [Ioffe and Szegedy, 2015]. Compared with it, LCIP only loses 0.21% of MAP on performance. We also re-conduct TCL using VGG-M according to the parameter settings in the paper (refer to Tab. 3). For the evaluation in ShapeNetCore55 dataset, we carry out our method (LCIP + softmax loss) on the more challenging perturbed version. For a fair comparison, we re-implement "TCL + softmax loss" on VGG-M which is indicated as TCL(VGG) in Tab. 2. We also choose Wang [Savva et al., 2016], Li [Savva et al., 2016], K-d network [Klokov and Lempitsky, 2017], MVCNN [Su et al., 2015] and GIFT [Bai et al., 2016] methods for comparison.

We take two types of results in the competition, namely macro and micro. As shown in Tab. 2, our method achieves the state-of-the-art performance.

4.2 Comparison with Other Loss Functions

To demonstrate the efficiency of our loss functions, we set various comparison experiments on ModelNet40 dataset. Besides the commonly used loss functions, we also carry out two well-known methods, coco loss and triplet-center loss, on VGG-M. The experiment results are shown in Tab. 3. Our comparison contains 3 parts:

The first is the comparison with commonly used metric loss functions. As a well-designed deep metric loss, LCIP dramatically improves the performance in MAP by 6.58% from softmax loss supervision. And our method outperforms TCL and coco loss which are improved efficient methods. Moreover, for the comparison of Lortho and Lortho,batch, we can see that they achieve similar results when combining with Lcluster. It also should be noted that the optimization will diverge if the CNN is trained solely with Lcluster or Lortho because both two terms are necessary.

The second part explores the combination with other loss functions. Both of our loss functions can combine softmax loss separately and the results show that the combinations bring a significant improvement. Compared with center loss, Lcluster achieves 81.94% in MAP when jointly trained with softmax loss, which is evidence that Lcluster has a comparable polymerization ability. And to prove the efficiency of Lortho, we conduct "Lortho + softmax loss". This combination brings a large improvement of 5.31% in MAP compared with single softmax loss, which means Lortho has a powerful capacity in enlarging inter-class distance.

Finally, we train the CNN with three loss functions and obtain an impressive performance. "LCIP + softmax loss" achieves a MAP of 87.08% and "LCIP + center loss" achieves a MAP 87.22% on Modelnet40. Note that the loss weight of softmax loss and center loss is 0.1 and 0.0003 respectively in the combination with our loss function. All these experiments demonstrate our loss design can generate robust and discriminative features.

𝜆𝜆

Figure 4: The retrieval performance (MAP) of different hyperparameter settings on ModelNet40.

4.3 Discussion

Sensitiveness of hyper-parameter. In Eq. 5, d is a constant value for stability. In our method, we choose d = 2 because this choice can reduce the sensitiveness of trade-off parameter λ. To investigate the influence of d and λ on the performance, we conduct experiments supervised by CIP loss on ModelNet40. The experiment results are shown in Fig. 4. We can see that CIP loss can obtain state-of-the-art results with different values of d by adjusting λ, but the sensitiveness of λ is different. The performance varies a lot when d = 1. With d = 2, CIP loss can converge stably with a wide range of λ, which means our loss design guarantees stability in the optimization.

Visualization of experiment results. We use t-SNE [van der Maaten and Hinton, 2008] to visualize the experiment results with different loss functions in Fig. 3. From the figure, we can see that under the supervision of CIP loss, the distance between classes is increased compared with softmax loss and center loss. Compared with TCL and coco loss, the intra-class variance is further decreased with CIP loss and the geometric separation become more clear.

5 Conclusion

In this paper, we propose a novel loss function named Collaborative Inner Product Loss with regard to large-scale 3D shape retrieval task. The Inner product is employed in the loss design which not only imposes a more strict constraint but also guarantees stability in optimization. The proposed loss function consists of Cluster loss and Ortho loss that play different roles: one for reducing the intra-class distance and the other for enlarging the inter-class margin, and both of them can combine with other commonly used loss functions. Experimental results on two large-scale datasets have proven the superiority of our loss functions.

Acknowledgements

This work is supported by the Beijing Municipal Natural Science Foundation (No.L182014), and the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No.VRLAB2019C05).