Learning by Actively Querying Strong Modal Features

1 Introduction

With the fast development of data collection techniques, complicated objects can be described by features from different data channels and are naturally with multi-modal feature presentations, e.g., modern mobile phones with different type of sensors can collect sensor signals from multiple channels. Recently, multi-modal learning techniques have been developed and paid more attentions to utilize the information from different modalities. Kiros et al. [2014] applied deep network to learn features over multiple modal data; Zhou et al. [2015] used the multi-modal time-series signals in mental health system; Nguyen et al. [2013] proposed the M3LDA to annotate image regions together with text tags, and provided a promising way to understand the relation between input patterns and output semantics. Meanwhile, different modalities are of various importance under specific circumstances, e.g., in medical tests for diagnosing the same disease, it is known that CT tests are with higher confidence, while on the contrary, X-rays are with lower confidence. We denote the modalities which can help the tasks more as strong modalities and on the contrary are the weak modalities. It is notable that strong modalities can lead to better performance, nevertheless, are more expensive for collection or extraction. Although existing modern multi-modal learning techniques can achieve better performance by incorporating with more strong modal features [Yang et al., 2015], they do not take the feature extraction expenses into consideration. Thus, how to reduce the informative strong modal feature extraction costs is an urgent problem.

⇤This research was supported by NSFC (61273301) and CCFTencent Open Research Fund (RAGR20150117).

Active learning aims at learning concepts by querying the labels of unlabeled data for better classification performance as well as reducing the costs of labeling. Settles [2010] designed a learning algorithm that connects active learning with multi-armed bandit; Zhong et al. [2015] added the "unsure" option for the crowd annotators in active learning; Huang et al. [2015] proposed a novel MLAL framework to query the relevance ordering of label pairs. Yet querying the ground truths is more expensive than collecting features from strong modalities. As a matter of fact, the ground truths can be regarded as a special type of "strong" modal features in certain extents, yet they are gathered by oracle labeling and are supposed with more expenses than strong modal feature collection or extraction by sensors.

In this work, we propose the ACQUEST (ACtive QUErying STrong modalities) strategy, which makes full use of strong modalities by actively querying strong modal features while reducing the feature acquisition expenses simultaneously following the style of active querying. Different from active learning, ACQUEST exploits multiple modalities by querying the features of "selected" instances from strong modalities rather than querying corresponding ground truths. At the same time, in ACQUEST, only the informative instances are selected to query strong modal features. It is notable that ACQUEST queries feature values of strong modalities without any interventions from oracle, and this makes ACQUEST can be trained with the least overall costs. An inverse prediction technique is also proposed and embedded for making the ACQUEST unified in one optimization formulation. The effectiveness of the ACQUEST is validated by extensive experiments. Section 2 is related work, our approach is presented in Section 3. Section 4 reports our experiments. Finally, Section 5 gives the conclusion.

2 RelatedWork

Multi-Modal learning has attracted many attentions [Wang et al., 2015], [Zhou et al., 2005], [Zhang and Li, 2014] and [Zhang et al., 2014]. Recently, some approaches distinguish the multiple modalities into strong modalities and weak modalities according to their importance for the concrete application requirements [Ion et al., 2006]. By incorporating strong modal information, better weak modal feature extraction can be performed and can consequently achieve better performance [Yang et al., 2015]. Nevertheless, these methods have not taken the expenses of strong modal feature extraction into consideration.

Active learning exploits unlabeled data by querying the labels of a subset of unlabeled instances for better classification performance, and can reduce the oracle labeling costs. The most popular active learning approaches usually choose the informative or representative instances for the ground truth querying [Sachan et al., 2015], [Fang et al., 2014], [Maria-Florina et al., 2007] and [Sanjoy and Daniel, 2008]. Huang et al. [2014] presented a novel active learning approach which considers both the informative and representative criteria simultaneously and achieved better classification accuracy while reducing the labeling costs.

Nevertheless, in multi-modal learning scenarios, two different facts induce further considerations on multi-modal feature collection and extraction:

• ÷ Strong modal features which help a concerned task more are with higher collection expenses than ordinary weak modal features, e.g., acquisition of finger prints needs specialized equipments;
• ÷ The expenses of gathering strong modal features are accomplished with sensors automatically and surely with less manual interactions, therefore are less expensive than directly querying the oracles.

To the best of our knowledge, previous multi-modal methods, which improved the classification performance using the auxiliary strong modalities, did not consider the collection expenses of strong modal features, while active learning methods query the labels of unlabeled instances to reduce the costs of labeling, yet cannot directly applied in the multi-modal learning scenarios. In this work, we focus on actively acquiring the most informative strong modal features from a portion of instances instead of the ground truths to ulteriorly reduce the data acquisition costs. The proposed approach ACQUEST (ACtive QUErying STrong modalities) selects the most informative and representative instances for strong modal feature values acquisition, and there is no need to query oracles for labels. Consequently, ACQUEST results in further reduction of the overall expenses on the collection of features or labels.

3 Proposed method

Suppose we have N examples, denoted by D = ¶(x1↪ y1)↪ (x2↪ y2)↪ · · · ↪ (xnl ↪ ynl )↪ xnl+1↪ · · · ↪ xN♦, the training dataset that consists of nl labeled data Dl and nu = N - nl unlabeled data Du, where each instance xi = [xi1 ↪ xi2 ↪ · · · ↪ xid ] 2 Rd , and yi 2 ¶-1↪+1♦ is the class label of xi. Meanwhile, in multi-modal learning, instance space can be denoted as, at least two parts without overlap, v = ¶v1↪ v2♦, where v1 2 Rd1 is raw features from weak modality and v2 2 Rd2 is the strong modal raw features, d = d1 + d2. In this paper, without any loss of generalities, each instance xi is denoted as (xi↪v1 ↪ xi↪v2 ).

3.1 Active Querying StrongModalities (ACQUEST)

Researchers claim strong modal features, which are with more discriminative abilities, meanwhile are with higher costs [Yang et al., 2015]. Yet it is a matter of fact that the ground truths are more expensive since collecting labels requires human efforts. As a consequence, in case of the connections between the strong modal features and the ground truth concepts are "idea" exploited, we can actively query the strong modal features of some informative and representative instances rather than true labels, to reduce the labeling expenses. This will result in less costs for feature value acquisition and is the basic idea of the proposed ACQUEST (ACtive QUErying STrong modalities). In this section, we focus on describing the novel approach in detail.

We start our discussion from a regularized classifier which can be generally formed as:

where f ⇤ can be a linear or kernelized classifier trained from labeled examples, H is the Reproducing Kernel Hilbert Space, `(z↪ ˆ z) can be any convex loss function. Given the classifier f ⇤ , the labels of unlabeled instances which are close to the decision boundary should be selected for ground truth querying in active learning, i.e., the selected instance xs↪v1 should lead to a small value for the object function regardless of its label ys.

In active learning scenarios, in order to select queries that are more representative [Huang et al., 2014], the evaluation function can be extended to include all the unlabeled data. If we know the class assignments yu 2 ¶◦1♦nu-1 for all unselected unlabeled instances in Du, the criterion of instances selection can be approximated by:

where the evaluation function can be represented as:

To select more informative example xs↪v1 , we expect that all unselected unlabeled instances from Du should result in a small value of L(Dl↪Du↪ yu↪ xs↪v1 ) in contrary to ys maximizing the value of L(Dl↪Du↪ yu↪ xs↪v1 ). We therefore approximate the solution for yu by minimizing L(Dl↪Du↪ yu↪ xs↪v1 ), which leads to the following surrogate for L(Dl↪ Du↪ yu↪ xs↪v1 ) in query selection:

where yu 2 ¶◦1♦nu-1 , ys 2 ¶◦1♦ and eventually we can reduce the labeling expenses using active queries on labels.

While in multi-modal learning scenarios, gathering ground truths should be more complicated, i.e., labeling multi-modal instances requires oracles inspecting more feature values from different modalities. However, by noticing that the expenses of collecting strong modal feature values (usually automatically collected by sensors or computers) are greatly less than that of labeling, we then can turn to query strong modal feature values instead of ground truths. ACQUEST approach is used to achieve this goal. Through explicitly figuring out the connections between strong modality and ground truth concepts, ACQUEST can integrate an inverse prediction technique to unify actively querying with multi-model learning in one optimization framework.

To simplify the discussion, we first model the connections between strong modal features xi↪v2 and weak modalities xi↪v1 with a linear model. It is assumed there can be an appropriate feature subspace on strong modality homologous with weak modal feature space: ˆxi↪v2 = g(xi↪v1 ) = xi↪v1 W, where the weak raw feature projection matrix W 2 Rd1⇥d2 . Note that this kind of connection can be easily extended to kernelized version further. The corresponding loss function describing the difference between strong modalities and transformed weak features is defined as: `0 (xi↪v2 ↪ g(xi↪v1 )). Here, `0 (z↪ ˆ z) can be any convex loss functions. Similar to the definition of ˆL(Dl↪ Du↪ xs↪v1 ), we have

where Xu↪v2 = [xnl+1↪v2 ↪ xnl+2↪v2 ↪ ▷ ▷ ▷ ↪ xN↪v2 ] 2 R(nu-1)⇥d2 is the unlabeled strong modal features except for xs↪v2 which is the strong modal feature values corresponding to the instance queried.

For simplicity, in our implementation, we set `(y↪ ˆ y) = (y - ˆ y)2 ◁2 and model the connection between strong modal features and ground truth concepts linearly, i.e., we assume yl = Xl↪v2 w, where the linear coefficients w 2 Rd2 . Xl↪v2 is the strong modal features of labeled examples. In this way, the connections between strong modal features and labels are actually in the form of least square minimization: argmin kXl↪v2 w - ylk22 , and of course has a closed form solution: the strong modal feature values can be represented as Xl↪v2 = ylw , where w is the pseudo inverse of w. By replacing the strong modal instance feature values Xv2 with yw , we can reform the equation above as:

It is notable that the closed form solution of Xl↪v2 = ylw provides an "inverse prediction" of strong modal features Xv2 with (pseudo) labels y and bridges the strong modalities and ground truths, which will further unify the ACQUEST in a holistic framework. In particular, to choose the most informative instances to query the strong modal feature values, we should consider both the ˆL(Dl↪ Du↪ xs↪v1 ) and ˆG(Dl↪ Du↪ xs↪v1 ) simultaneously, and eventually the query of ACQUEST is made according to the following criterion:

and the instance need to be queried can be obtained by:

where Fv1 = ¶fv1 (x1↪v1 )↪ fv1 (x2↪v1 )↪ ▷ ▷ ▷ ↪ fv1 (xN↪v1 )♦ 2 RN is the predictors for features of weak modality, and Gv1 = ¶gv1 (x1↪v1 )↪ gv1 (x2↪v1 )↪ ▷ ▷ ▷ ↪ gv1 (xN↪v1 )♦ 2 RN⇥d2 is the extracted features from weak modality. y = [y1↪ y2↪ ▷ ▷ ▷ ↪ yN] 2 RN , where for very limited number of the labeled data, yi = 1 if xi belongs to positive class, and yi = -1 is negative. For unlabeled instances, yu equals to 1 or -1. It is notable that in Eq. 2, the strong modal features are represented by yu and w with the "inverse predictions" treatments which simplifies the enumeration of feature values for strong modalities.

kfv1 k2H and kgv1 k2H are the structure risk of predictor fv1 and feature extractor gv1 in each function space respectively. In order to predict with the lower-cost weak modal information only in test phase, we simply assume a linear predictor defined in the extracted feature space of weak modalities, i.e., Fv1 = Xv1 Wv, where v 2 Rd2 . As a consequence, the definition of ˆ A(Dl↪ Du↪ xs↪v1 ) can be reformed as:

where the weak modal instances Xv1 can be projected into the strong modal feature space and represented as Xv1 W, therefore the 4th term reduces the differences between strong modal features and projected weak features. Wv is the predictor for Xv1 , consequently ky-Xv1 Wvk2F can act as linear classifier on weak modality. Note that different from [Yang et al., 2015], ACQUEST can reduce the strong modal feature costs on both test and training phase, while the former needs all strong modal features during training.

4 Solution to Training ACQUEST Model

In this section, we further derive the training approach for selecting the strong modal features of the instance which should be queried. When least square loss is utilized, it is straightforward that

where Lv1↪l = (λ1I +X> v1 Xv1 )-1 . As a consequence, one part of target function ˆL(Dl↪Du↪ xs↪v1 ) can be simplified as:

Similarly, for the classification model trained by the strong modal examples, we also can have:

where Lv1↪v2 = (λ2I + X> v1 Xv1 )-1 and ˆy = yy>Xv2 (X> v2 Xv2 )-1 , and ˆG(Dl↪Du↪ xs↪v1 ) can be simplified as:

According to Eq. 2, we have:

Query the most representative and informative instance needs to compute the evaluation function ˆ A(Dl↪ Du↪ xs) in Eq. 4 for every unlabeled instance xs↪v1 . Note that the calculation can be further reduced or simplified since

where L can be Lv1↪l or Lv1↪v2 . Lm↪n is the sub-matrix of L with corresponding rows and columns according to subscripts m and n, and the above objective is concave in ys and convex in yu, the minimization on yu and maximization on ys can be switched. Moreover, the solution to min yu y> u Lyu can be

obtained in a closed form:

By substitute ˆyu to ˆ A(Dl↪ Du↪ xs), we consequently have

Thus in ACQUEST training phase, we need to evaluate the criterion in Eq. 5 to select the instance for strong modal feature values querying according to Eq. 2, this procedure should be repeated for unlabeled instances iteratively.

In each iteration, one most valuable unlabeled instance is selected for strong modal feature value acquisition. Once we have obtained the strong modal features of the most valuable instance, we can have the parameters W and v updated. Note that the 2nd term in Eq. 3 involves the product of weak modal feature extraction matrix W and the weak modal predictor v, we can use the gradient descent techniques for updating the parameters. Here alternative descent algorithm is used in this work, since this will lead to closed form alternating and finally simplify the calculations.

Fix W, Update v

When W is fixed, note that the 3rd and 4th term in Eq. 3 are not related to v, therefore it can be equivalently written as:

Eq. 5 has a closed-form solution for v:

Algorithm 1 The ACQUEST Algorithm

Require: Xl↪v1 , Xu↪v1 , Xl↪v2 , Xu↪v2 , λ1, λ2, y, max-iter;

• 1: repeat
• 2: for i = 1 to nu do
• 3: Calculate ˆ A(Dl↪ Du↪ xs↪v1 ) using Eq. 5
• 4: end for
• 5: Select the xs⇤ with the smallest ˆ A(Dl ↪ Du↪ xs↪v1 )
• 6: Calculate the pseudo label ys⇤ using y = sign(xs↪v2 w)
• 7: Xl↪v1 =Xl↪v1 [ (xs⇤↪ ys⇤); Xu↪v1 =Xu↪v1 ∩ xs⇤
• 8: Training classifier with closed solutions of Eq. 5 and 6
• 9: until the query number (number of iterations) exceeds max-iter.

Fix v, Update W

When v is fixed, Eq. 3 can be equivalently written as:

Obviously, Eq. 6 also has a closed solution for W.

Prediction with only weak modality required

In the prediction stage, we can predict the test instances which are represented with weak modalities only (i.e., no strong modal features are provided), with the obtained W and v in training phase in following equation:

where xi↪v1 is weak feature values of a test instance. The bias b for predictors is obtained during the training stage, and the instance label should depend on sign(fv1 (xi↪v1 )) . The pseudo code of ACQUEST is shown in Algorithm 1.

5 Experiment

Datasets and Configurations

In this section, we introduce the compared methods and datasets before giving the empirical results of ACQUEST. ACQUEST can be adopted for many applications where there are multi-modal features. In this paper, 12 image datasets are used in our empirical investigations. A subset of NUS [Chua et al., 2009] contains 9,109 images of 10 categories, and 6 groups of features extracted. MSRA [Wang et al., 2009] subset contains 10,680 images of 9 categories, and 7 groups of features are extracted. Animal [Christoph et al., 2009] (represented as ANIM in short in the following content) subset contains 30475 images of 50 animals classes, and 6 pre-extracted feature representations are extracted for each image. All the feature sets can be separated into strong modal features and weak modality. More specifically, for NUS the color histogram features are selected as weak modality while the rest are strong modal features, and in NUS, 4 subsets are constructed for balanced binary classification, i.e., lake vs. railroad, surf vs. map, map vs. boats and reflection vs. boats (denoted as NUS1, NUS2, NUS3 and NUS4 respectively). Similarly, for MSRA, HSV color histogram is the weak modal features and the rest are strong modal features, and 4 balanced subsets are selected for binary classification, denoted as MSRA1, MSRA2, MSRA3 and MSRA4 respectively. For ANIM, PyramidHOG (PHOG) features are weak modal features and the rest are strong modal features. Besides, 4 subsets are also constructed from ANIM, i.e., horse vs. cow, rhinoceros vs. otter, rhinoceros vs. collie and rhinoceros vs. raccoon (denoted as ANIM1, ANIM2, ANIM3 and ANIM4). The judgement of weak/strong modalities are made according to the feature extraction time costs of each group modalities.

For all datasets, 66% instances are randomly picked up for training, and the remains are used as test set. The labeled ratio is set to 10% for training set. All experiments are repeated for 30 times. During the training phase, at most 30 unlabeled instances are automatically selected for strong modal feature value querying in each iteration. More specifically, in ACQUEST an unlabeled data instance is first selected for strong modal feature values querying according to the criterion discussed in Eq. 5, followed by the calculations of the corresponding pseudo labels, and then the classifier is retrained using this corresponding instance with both strong modal features and its pseudo label. The avg. and std. of predictions are recorded for evaluation. In all experiments, the parameters λ1 and λ2 in the training phase are tuned in {10-1 ↪ 1↪ 10 . Empirically ACQUEST converges when the difference of the objective value of Eq. 3 is less than 10-5 .

Since ACQUEST actively queries the strong modal feature values, we compare it with 5 active learning algorithms and a recently proposed semi-supervised multi-modal learning approach [Yang et al., 2015] which also takes strong/weak modalities in consideration. For 5 compared active learning algorithms, both strong and weak modal features are provided and these active learning models are trained with ground truths while queries are made. The 6 compared methods are: RANDOM: active learning with randomly selected queries;

MARGIN: margin-based active learning, linear classifier as the base learner [Tong and Koller, 2002];

IDE: active learning that selects informative and diverse examples [Jin et al., 2008];

DUAL: a dual strategy for active learning that exploits both informativeness and representativeness for selection [Pinar et al., 2007]. Parameter of DUAL is set as k = 1;

QUIRE: active learning algorithm which queries unlabeled instances that are both informative and representative;

ARM: multi-modal learning approach which improves multimodal learning performance via extracting the most discriminative weak modal feature subspace with the help of strong modal information.

Classification Accuracy Comparisons

Table 1 records the accuracies (avg.◦ std.) of the ACQUEST and compared methods, the number of queries is 30 for active learning methods and ACQUEST. For each dataset, the best result is highlighted in bold, ACQUEST is tested with fv1 on weak modality only.

From Table 1, it clearly reveals that on 8 real world datasets, the average accuracies of ACQUEST are the best, and for the rest datasets, i.e., NUS4, MSRA2 and ANIM4, ACQUEST achieves the runner-up.

Comparisons when Number of Queries Changes

To investigate the performance of compared active learning methods when number of queries changes, we conduct additional experiments and record more results. Table 2 summarizes the win/tie/loss counts of ACQUEST versus active learning methods with t-test at significance level 95%. The number of queries varies from 5 to 30. From Table 2, the win/tie/lose counts clearly show that ACQUEST approach is superior to those compared active learning methods no matter what the number of queries is.

A detailed classification performance on different number of queries are recorded in the Fig. 1. As in Table 2, the number of queries is set from 5 to 30. From these subplots in Fig. 1, it can be observed that the performance (accuracy) is increased faster for ACQUEST than other compared methods, e.g., on most datasets, ACQUEST gets a high accuracy within 10 queries; besides, the performance of ACQUEST is generally better than other compared approaches on most datasets when the number of queries increases. Moreover, after 10 queries, it can be found that ACQUEST achieves a stable performance on most datasets. Yet the performances of all compared methods are unstable as the queries increasing on some datasets (e.g., MSRA1, MSRA3, ANIM1, ANIM3), this phenomenon can be addressed to the fact that strong modal features, which are picked according to feature extraction costs, are not real "strong" modal features, and perhaps cannot replace the ground truths perfectly for the disturbance of feature noises. Consequently, the pseudo labels from the inverse prediction might be less reliable on these datesets.

6 Conclusion

In this work, feature value acquisition expenses for "strong" modal features are considered in both the training and test phase of multi-modal learning. We proposed a new active strong modal feature values acquisition approach called ACQUEST. ACQUEST can exploit the strong modal information by querying the corresponding feature values of selected instances rather than querying the labels directly as in active learning. Due to the fact that feature acquisition hardly needs oracle interventions, ACQUEST requires less human resources, and according to the experiments, our approach achieves comparable or even better performances. It is notable that, different from existing multi-modal learning approaches which take strong/weak modalities in consideration, ACQUEST actively selects the instance for strong modal feature acquisition automatically, i.e., ACQUEST can consequently reduce the expenses of strong modal data collecting and labeling in both the training and test phase, while other strong/weak modal learning approaches focus on reducing expenses in test phase. An inverse prediction technique is also proposed for unifying ACQUEST in one optimization formalization as well, which simplifies query selection. Empirical results clearly validate its effectiveness on feature querying, expenses reducing etc. How to identify the real "strong" modality in multi-modal scenarios and extend ACQUEST for multi-class applications should be interesting future works.