Multi-View Multi-Label Learning with View-Specific Information Extraction

1 Introduction

In many real-world applications, objects are with diverse representations and rich semantics simultaneously. For example, in image analysis, a natural scene image can often be represented by its visual features such HSV color histogram, globe feature (Gist) and scale invariant feature transform (SIFT), while it may be annotated with labels {sky, water, cloud}. In video annotation, the representations of a film e.g. Avengers: Infinity War are usually from multiple channels of information such as text description, audio, cover picture, frame extraction, meanwhile it can be annotated with action movie (type), America (country) and Anthony Russot and Joseph Russo (directors). The main challenge of such problems is how to integrate the multiple types of heterogeneities in an efficient and general way. Multi-view multi-label learning serves an important framework to solve the above problem.

∗Corresponding author

Formally speaking, let X = Rd1 ×Rd2 · · ·×Rdk be the feature space of k views representations, where dv(1 ≤ v ≤ k) is the feature dimension of v-th view. Let Y = {yj}qj=1 be the label space with q class labels. Given the training set D = {(xi, Yi)|1 ≤ i ≤ m}, where xi ∈ X is the feature vector [x1i , x2i , · · · , xki ] and Yi ⊆ Y is the set of relevant labels associated with xi. The task of multi-view multi-label learning is to learn a predictive model h : X → 2Y from D which can assign a set of proper labels for the unseen instance.

There have been some attempts for multi-view multi-label problem. In [Xing et al., 2018], the authors introduce a predictive reliability measure to select samples which are used to share label information with other views in the co-training manner. [Zhu et al., 2016; Liu et al., 2015; Luo et al., 2013] focus on the issue of multi-view image classification and map each view into a shared space to eliminate noise and redundancy while maintaining sparse, low-rank and manifold structure of image data respectively. [Zhu et al., 2018; Zhang et al., 2018] aims to further remain consensus on multi-view latent spaces by using Hilbert-Schmidt independence criterion during the mapping procedure.

Nonetheless, the above methods suffer from the following two different issues. Firstly, when trying to find shared information among all views to eliminate noise and redundancy, a common practice is to map each view into a shared subspace. However, this procedure is always done in an independent way, i.e. there is no communication among various views. It is hard to ensure that common semantic information is fully tapped. Secondly, the underlying fact that each view contains their own specific contribution to the multi-label prediction, is ignored. For example, a picture of pink rose may be tagged with two labels {pink, flower}, while its representation can be described by HSV and Gist. We can easily find the correspondence between them, i.e. pink is described by HSV view feature, while flower is described by Gist view feature. However, existing methods only try to find the shared information between HSV (color) and Gist (texture), while it is more reasonable to consider extracting their own specific information.

To deal with the above two issues, we present a novel multi-view multi-label neural network framework, which we call SIMM (view-Specific Information extraction for Multiview Multi-label learning), to leverage shared subspace exploitation and view-specific information extraction. Firstly, SIMM jointly minimizes confusion adversarial loss and multi-label loss to utilize shared information from all views. Secondly, SIMM enforces an orthogonal constraint w.r.t. the shared subspace to utilize view-specific discriminative information. Finally, the shared and view-specific information are synergized to learn the semantics. Extensive experiments on real-world multi-view multi-label data sets clearly show that SIMM achieves highly competitive performance.

The rest of this paper is organized as follows. Section 2 presents technical details of the proposed approach. Section 3 discusses existing works related to SIMM. Section 4 reports detailed results of comparative experiments. Finally, Section 5 concludes.

2 The ProposedMethod

The goal of multi-view multi-label learning is to fully integrate various representations of a single object and assign the proper rich semantics to it. As mentioned above, information from different views usually contains the shared and specific parts. Thus, two key steps of SIMM are shared subspace exploitation and view-specific information extraction.

SIMM implements those two key points by using a neural network framework. Considering the network, the overall loss function naturally turns out to be the following form:

where α and β are trade-off factors that control the interaction of the loss terms. Final multi-label classification loss Lml controls the model to predict the output labels. Let yj i denote the ground-truth label of xi on the j-th label. yj i = 1 if j-th label is the relevance label, yj i = 0 otherwise. Let ˆ yj i denote the prediction output. The Lml can be calculated as follows:

2.1 Shared Subspace Exploitation

In traditional multi-view learning, the shared subspace exploitation is often implemented in an independent way. Thus, the communication among all views is neglected. Inspired by [Liu et al., 2017], SIMM implements the mapping in a confusion way by minimizing the adversarial loss Ladv. In particular, SIMM aims to confuse the discriminator that the coming shared subspace representation belongs to which view.

Let cv be the l dimension shared subspace representation of original v-th view feature xv . It is extracted by a shared subspace extraction layer H, i.e. cv = H(M v (xv )), where M v (·) is used to map each v dimension original feature vector into the same l dimension. Let zi be the k dimension view label vector of cvi , where only zv i equals to 1 while others equal to 0. This indicates that cvi comes from the v-th view. A training set Dadv = {(cvi , zi)|1 ≤ v ≤ k, 1 ≤ i ≤ m} can be constructed for discriminator D. Let ˆ z be the prediction output by D, i.e. ˆ zi = D(cvi ), Ladv can be formed as:

where F(·) should be a monotonically decreasing function 1 . In this way, we confuse the discriminator so as not to recognize the true view of the coming shared subspace representation, which indicates that there is no special information in it. In other words, the incorporated representation cv only contains the shared information of xv .

However, it might be problematic if only Dadv is used as noise can also confuse the discriminator easily. Therefore, we use Lsml (Shared subspace Multi-Label Loss) to guarantee that cv contains certain semantics. In particular, a training set Dsml = {(cvi , yi)|1 ≤ v ≤ k, 1 ≤ i ≤ m} can be constructed for shared subspace representation prediction layer Gs. Let ˆ y′ be the prediction output byGs, i.e. ˆ y′v i = Gs(cvi ). Lsml can be formed as:

1 In this paper, F(x) = e-x

The shared subspace exploitation by SIMM is no longer conducted in an independent way as H and D meet the information from all views with Gs maintaining semantics. Combining the above losses together, Lshared can be written as:

2.2 View-Specific Information Extraction

In SIMM, helpful view-specific information is further considered. Actually, it is difficult to define what is the specific information of a particular view. However, we can extract it from original information by eliminating shared information. This can be implemented by enforcing an orthogonal constraint. Let sv be the l dimension feature vector extracted by specific information extraction layer Ev , i.e. sv = Ev (xv ). Let c be the l dimension feature vector containing shared information among all views. c is performed by element-wise addition on each individual cv , i.e. c = ∑kv=1 cv . The Lspecific encourages orthogonality between sv and c:

where ‖·‖22 is the squared L2-norm. Lspecific encourages sv extracted from the original v-th view feature vector xv to be as different from c as possible.

2.3 TheWhole Framework

The whole framework of SIMM is shown in Figure 1. The goal of training is to minimize the whole loss function (1) with respect to parameters Θ = {θE , θM , θH , θD, θGs , θG}. Each unit is a neural network framework and we use popular optimization algorithm Adam [Kingma and Ba, 2015] to do the back propagation to update Θ simultaneously in the training phase. In the testing phase, given an unseen instance x∗ , its prediction output ˆ y∗ can be obtained as:

where c = ∑kv=1 H(M v (x∗v ))

3 RelatedWork

In the past years, multi-label learning has been widely used in many kinds of applications, such as text categorization[Ueda and Saito, 2003], bioinformatics [Zhang and Zhou, 2006], web mining[Tang et al., 2009], etc. Following [Zhang and Zhou, 2014], existing multi-label methods can be categorized into two groups, i.e, problem transformation methods and algorithm adaptation methods. Problem transformation methods aim to tackle multi-label learning problem by transforming it into other well-established learning scenarios, such as Binary Relevance [Boutell et al., 2004] transforms the task of multi-label learning into the task of binary classification while Calibrated Label Ranking [F¨urnkranz et al., 2008] transforms the task of multi-label learning into the task of label ranking. Algorithm adaptation methods tackle multi-label learning problem by adapting popular learning techniques to deal with multi-label data directly, such as ML-kNN [Zhang and Zhou, 2007] adapts lazy learning technique while MLDT [Clare and King, 2001] adapts decision tree technique.

In multi-view learning, the information in some views can help handle the weakness of other views. Furthermore, multiview learning can be embedded into multi-label learning tasks naturally to further improve the classification performance. The most important part in multi-view multi-label learning is how to utilize and communicate heterogeneous information among all views under the multi-label framework. [Xing et al., 2018] focuses on selecting reliable samples from one view and passing them to other views. However, the communication among views is only on the label level. [Zhu et al., 2016] introduces a block-row regularizer to reduce the noise and redundancy. The method in [Liu et al., 2015] seeks a low-dimensional common representation of all views and constructs the classifier based on matrix completion. [Luo et al., 2013] integrates multiple features under multi-view vector-valued manifold regularization. [Zhu et al., 2018; Zhang et al., 2018] try to remain latent semantic when studying the low-dimensional common representation.

However, for most existing methods the mapping from original views to the shared subspace is conducted in an independent way where there is only limited communication among each view. Furthermore, the specific characteristics of individual view is ignored. SIMM serves as the first attempt towards jointly enhancing communication during shared subspace exploitation and remaining view-specific information.

4 Experiment

4.1 Experimental Setting

Data Sets

A total of eight multi-view multi-label data sets are employed for performance evaluation including six benchmark data sets 2 and two real-world video annotation data sets.

Emotions has two feature views: 8 rhythmic attributes and 64 timbre attributes. Yeast also has two views including the concatenation of the genetic expression (79 attributes)

2 Publicly available at http://mulan.sourceforge.net and http://lear.inrialpes.fr/people/guillaumin/data.php

and the phylogenetic profile of a gene (24 attributes). For Corel5k [Duygulu et al., 2002] and EspGame [Von Ahn and Dabbish, 2004], we choose four representative feature views: DenseHue, Gist, DenseSift, HSV, whose dimensions are 100, 512, 1000 and 4096 respectively. For Pascal [Everingham et al., 2010] and Mirflickr [Huiskes and Lew, 2008], Tag view is further utilized as the textual view besides the above four views [Guillaumin et al., 2010], whose dimensions are 804 and 457.

Youku data set consists 148,089 videos in 114 classes collected from the Video Application YOUKU. Each object has four views: textual description in the video title, the audio information, cover picture and video frame extraction. The dimensions are 100, 128, 2048, 2048 respectively. For textual view, we extract the weights of each word in the title. Using the pre-trained word embedding, title embedding vector can be obtained by weighting the word embedding. For audio view, we follow [Gemmeke et al., 2017] and reduce dimension by using Principle Component Analysis (PCA). For image view, we use Inception-v3 proposed in [Szegedy et al., 2016]. For video view, we extract video frames every two seconds and use the same procedure as image view to obtain frame-level features. Then, these frame-level features are aggregated into video-level by mean-pooling and L2-normalization. Label information is obtained by manual labeling, from which we choose the most frequent 114 ones.

Given the data set D, we use |D|, V (D), VDim(D), CL(D) to represent its number of samples, number of views, dimension of each view, class labels. Moreover, several other properties of multi-label data sets are denoted as [Zhang and Zhou, 2014]:

Label cardinality: LCard(D) = 1N ∑Ni=1 ∑qj=1 I(yij = +1) which counts the average number of relevant labels per example;

Label density: LDen(D) = LCard(D) CL(D) which normalizes label cardinality by the number of class labels;

Distinct label sets: DL(D) = |{y|∃x : (x, y) ∈ D}| which counts the number of distinct label vectors (relevant label set) existing in D;

Proportion of distinct label sets: PDL(D) = DL(D) |D| which normalizes DL(D) by the number of examples.

Table 1 summarizes detailed characteristics of the experimental data sets.

Comparing Algorithms

The performance of SIMM is compared against six algorithms, including two baselines related to SIMM, a multi-label algorithm ML-KNN with two types of feature inputs and two multi-viewmulti-label algorithms.

Benchmark: A multi-layer perceptron where the number of parameters is no less than SIMM. The input is the concatenation of all view features.

COMMON: The prediction results are only obtained by shared information. i.e. ˆ y∗′ = 1k ∑kv=1 ˆ y∗′v

ML-KNN(C) &ML-KNN(B) [Zhang and Zhou, 2007]: A lazy multi-label algorithm with two types feature inputs. (C) stands for the concatenation of all views. (B) stands for the best performance obtained from the best single view. [configuration: k = 10]

F2L21F [Zhu et al., 2016]: A multi-view multi-label algorithm using block-row regularizer to reduce the overlap, noise, and redundancy in multiple views. [configuration: λ1 = 10, λ2 = 10]

LSAMML [Zhang et al., 2018]: A multi-view multilabel algorithm using Hilbert-Schmidt Independence Criterion to enhance the dependence between different pairs of views. [configuration: r ∈ {2, 3, 4, 5}, β and γ ∈ {0.01, 0.1, 1, 10, 100}]

For SIMM, in order to make the model more elegant and lightweight, we set each module to be only a fully connected layer without hidden layer. l is fixed to 64. In light of comparison to COMMON, α is fixed to 1. β is selected from {0.1, 0.01, 0.001, 0.0001}.

Experimental Protocol

In this paper, six widely-used multi-label metrics are employed for performance evaluation, including Hamming Loss, Average Precision, One Error, Ranking Loss, Coverage and Micro-F1, which consider the performance of multi-label predictor from various aspects. Concrete metric definitions can be found in [Zhang and Zhou, 2014], and Coverage metric is normalized by the number of class labels (i.e. q). For Average Precision and Micro-F1, the larger the values the better the performance. For the other four metrics, the smaller the values the better the performance.

For each data set, ten-fold cross-validation is performed where the mean metrics results and standard deviations are recorded for all comparing approaches.

4.2 Experimental Results

Detailed experimental results are reported in Table 2, where the best performance on each data set is shown in boldface. To analyze the relative performance among the comparing approaches in a systematic manner, Friedman test is employed as the statistical test for performance comparison.

Table 3 reports Friedman statistics FF and the corresponding critical values in terms of each evaluation metric. It is obvious that the null hypothesis of equal performance is rejected at 0.05 significance level. Accordingly, post-hoc BoferroniDunn test is performed to compare the relative performance among the comparing approaches. The CD diagrams are presented in Figure 2, where the average rank of each approach is marked along the axis (the smaller the better).

Based on the reported experimental results, the following observations can be made: (a) Among the 48 configurations (8 data sets × 6 evaluation metrics), SIMM ranks 1st and 2nd in 87.5% and 10.4% cases respectively. (b) It is remarkable that SIMM achieves best performance in all cases on Hamming Loss, Ranking Loss and Coverage. (c) It is noticeable that SIMM outperforms all the comparing algorithms on Emotions, Pascal, Mirflickr and Youku15w on all metrics while outperforms all the comparing algorithms on Espgame and Youku3w on all metrics except on Micro-F1.

4.3 Further Analysis

In order to examine the effectiveness of Lshared and Lspecific, we remain the basic structure of our model and remove these two constraints by setting α and β to zero. Figure 3 shows the comparison results on Pascal and Youku15w data sets. The curves show the change of the metric results with the number of epoch. It is clearly shown that the model performed worse without Lshared and Lspecific (SIMM-NL). In other words, it is not enough to fuse the information from different views directly. SIMM is a good help to separate the shared and specific information.

5 Conclusion

In this paper, a novel neural network based approach is proposed to solve the multi-view multi-label problem. Specifically, we enhance the communication among views while remain individual specific characteristics through two steps, i.e. shared subspace exploitation and view-specific information extraction. Effectiveness of the proposed approach is validated via comprehensive experiments on real-world multi-view multi-label data sets. In the future, it is interesting to extend the structure of each neural network module. Meanwhile, note that sv and c are fixed with the same dimension due to the orthogonality constraint and therefore a more general way of specific information extraction can be explored.

Acknowledgments

This work was completed while Xuan Wu was conducting her internship at Alibaba Group. The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Key R&D Program of China (2018YFB1004300), the National Science Foundation of China (61573104), and partially supported by the Collaborative Innovation Center of Novel Software Technology and Industrialization.