Object Recognition with Hidden Attributes

1 Introduction

Object recognition in computer vision generally refers to the recognition of object images into different categories such as "bird", "aeroplane", "bicycle", etc. In recent years, computer vision researchers explore to assign a list of attributes [Farhadi et al., 2009] to the object images. These attributes [Ferrari and Zisserman, 2007] are manually specified and semantically meaningful descriptions about the object shape (e.g. "is cylindrical"), parts (e.g. "has head", "has leg"), materials (e.g. "made of wood"), color (e.g. "is red"), etc. The approaches [Farhadi et al., 2009; Lampert et al., 2009; Wang and Mori, 2010; Parikh and Grauman, 2011a; Kovashka et al., 2011]) that utilize the assigned attributes to benefit object recognition can be called attribute-based object recognition.

Most existing attribute-based object recognition approaches (e.g. [Farhadi et al., 2009; Lampert et al., 2009;

Figure 1: An example of attributes for two animals, where the bird has "beak", "wing" and is covered with "feather", and the cow has "ear", "snout" and is "furry".

Wang and Mori, 2010; Parikh and Grauman, 2011a; 2011b; Kovashka et al., 2011]) utilize attributes as an intermediate layer in the classifiers cascade. In the testing phase of these approaches, attributes are first predicted by the pre-trained attribute classifiers. And, these predicted attributes are further utilized by the attribute based object classifiers as midlevel input for object recognition. Typical applications of these approaches include zero-shot transfer learning [Lampert et al., 2009], description of unfamiliar objects [Farhadi et al., 2009], and improving the object classification [Wang and Mori, 2010], event recognition [Wang and Ji, 2012], and phone recognition [Zhao et al., 2015].

However, due to tremendous variations in vision applications, attribute recognition itself is challenging. Moreover, poor quality attribute measurements in the middle level would adversely affect the subsequent object classification. This dilemma motivates us to avoid utilizing attributes as middle level representation, and to explore incorporating attributes in a different way, where we have access to ground truth attributes during training, but do not utilize the predicted attributes explicitly or implicitly for final stage recognition during testing. In this paper, we call these attributes that are available only during training as hidden attributes. We hope the hidden attributes utilized in our approach can still improve the object recognition.

This hidden attribute setting lies in the "learning with hidden information" (LHI) paradigm [Vapnik and Vashist, 2009]. In this paradigm, the hidden information a are utilized only during training. They help learn a better classifier (e.g. linear classifier y = sign(w>x)) from feature x to label y that can outperform the traditional classifier (e.g. y = sign(w> 0 x)) learned without hidden information. Hence, in this paradigm, the hidden information serves only for the purpose of obtaining a better parameter vector w that is in the same dimension as the original parameter vector w0. Compared to the traditional mid-level based attribute approaches, this paradigm can avoid the propagation of erroneous attribute predictions to the subsequent object classification.

In this paper, we propose two novel formulations to incorporate hidden attributes during model learning. Our first approach (xLR+) utilizes hidden attributes as additional information to improve the target model that predicts object label y from image feature x. In the second approach (LRRel+), we further incorporate the semantic relationships between the objects and the hidden attributes. Finally,we further combine both formulations as regularization terms into one unified learning objective (xLR-Rel+) that receives the best performance.

In summary, the major contributions of this work include: 1) we propose to incorporate hidden attributes for object classification; 2) we propose the formulations including xLR+, LR-Rel+ and xLR-Rel+ that use hidden attributes as extra information and exploit their relationships with objects.

2 RelatedWork

Utilizing attributes to enhance the object recognition performance has drawn great attention in recent years. Work in this area can be divided into sequential attribute and object recognition, and joint attribute and object recognition. The sequential approaches [Farhadi et al., 2009; Lampert et al., 2009; Parikh and Grauman, 2011a; 2011b; Akata et al., 2013; Wang and Ji, 2014] utilize attributes as an intermediate representation between low-level image features and high-level categories. [Farhadi et al., 2009] use linear classifiers like SVM to predict attributes from shared image features, and then use the predicted attributes for object categorization. However, the sequential approaches still require to train attribute classifiers with training data, and then predict the attribute labels or infer attribute scores during testing. The performance of these methods is therefore subject to the performance of the attribute classifiers. To address this problem, the joint approach performs attribute and object recognition simultaneously in order to exploit their interdependencies. Wang and Ji [Wang and Ji, 2013] utilize a Bayesian network (BN) with learned structure to improve both attribute prediction and object recognition with captured attribute relationships. Also, the multi-task learning approach in [Hwang et al., 2011] simultaneously learns multiple classifiers for object recognition and attribute prediction tasks based on the shared feature assumption. The joint approaches need learn attribute and object classifiers simultaneously and they are therefore computationally complex. In contrast, our approach focuses only on the object recognition task.

Recently, [Wang et al., 2014] propose to incorporate hidden information for learning logistic regression classifier LR+. While its formulation looks similar to the first formulation in this paper, our work significantly differs from the work in [Wang et al., 2014]. The work in [Wang et al., 2014] studies learning LR+ using hidden information as extra information. Comparatively, our work focuses on object recognition with hidden attributes. We incorporate hidden attributes as extra information in xLR+, exploit the object-attribute relationships in LR-Rel+, and further propose a combined formulation xLR-Rel+. Both LR-Rel+ and xLR-Rel+ are completely different from the LR+ in [Wang et al., 2014]. Our xLR+, LR-Rel+, and xLR-Rel+ approaches all outperform the LR+ approach by [Wang et al., 2014] in the experiments.

3 Object RecognitionWith Hidden Attributes

We first define both the traditional supervised object recognition and the the proposed object recognition with hidden attributes.

Traditional supervised object recognition can generally be formulated as: given a set of N labeled training samples represented by image feature vector set X = ¶x1↪ · · · ↪ xN♦ ⇢ X ⇢ Rd , and the object label set Y = ¶y1↪ · · · ↪ yN♦ 2 Y with Y = ¶-1↪ 1♦ for binary cases, learn a mapping function f : X 7-! R with parameter w from the function space F of all possible functions (e.g. all linear functions f(x) = w>x) to predict the object label y from the input image feature x as accurate as possible. Generally, the object classifier parameter w can be learned by minimizing the objective function shown in Equation 1, where l(yi↪ xi;w) is the loss function and ♣♣w♣♣22 is a regularization term to avoid overfitting.

Object recognition with hidden attributes differs from the traditional supervised object recognition problem in that additional hidden information vectors (i.e., the ground truth attributes in this paper) A = ¶a1↪ · · · ↪ aN♦ ⇢ A are also provided for each training sample, where each M dimensional vector ai corresponds to the training sample pair (xi↪ yi). Object recognition with hidden attributes can hence be stated as: given N labeled training triplets ¶(xi↪ ai↪ yi)Ni=1 ♦, learn a mapping function f 0 : X 7-! R, with parameters w0 from the same function space F of all possible functions (e.g. all linear functions f(x) = w>x) to predict object label y from input image feature x as accurate as possible.

Following this definition, the new mapping function f 0 : X 7-! R does not depend on hidden attribute space A, but hidden attributes will influence parameter w0 in training. We expect w0 to be better than w in predicting y from x.

3.1 Hidden Attributes as Extra Information

In the object recognition with hidden attribute setting, hidden attributes can be utilized as additional information to learn a mapping function g : A 7-! R with parameter w⇤ for the prediction of class label y. In this paper, we call the g mapping function with parameter w⇤ as the hypothetic model. And also, the f function f : X 7-! R could be called as the target model.

As shown in Figure 2, during training, the hypothetic model and target model share the same object class labels ¶yi♦i=1↪▷▷▷↪N, but the information input for the hypothetic model is ¶ai♦i=1↪▷▷▷↪N which is the ground truth attributes.

Since both the image feature x and the hidden attributes a describe the object for each training sample, our basic idea in this formulation is to link the hypothetical model and the target model by regularizing the prediction score f(x;w) of the target model to be close to the prediction score g(a;w⇤ ) of the hypothetical model. We denote this regularization as the dissimilarity regularization.

Suppose the loss functions of the target model and hypothetic model for the ith training sample to be represented by l(yi↪ xi;w) and l(yi↪ ai;w⇤ ) respectively. Under the dissimilarity regularization, we would learn the parameters w and w⇤ of two models simultaneously with the dissimilarity regularization term, and the two parameter regularization terms kwk22 and kw⇤ k22 incorporated as shown in Equation 2:

where λ is a positive coefficient for the dissimilarity regularization term, γ1 and γ2 are the positive coefficients for the corresponding parameter regularization terms, and ⌘ is the positive weight on the loss function of the hypothetic model. With this objective function, our general approach would not only minimize the loss functions for supervised learning of both models, but also minimize the dissimilarity between the predictions of two models on the original feature space and the hidden information space respectively. The regularization term effectively ties the learning of the target classifier to that of the hypothetical model such that the hidden attributes a can influence the parameters of the target model.

The general approach described in Equation 2 incorporates a squared difference term which is similar to the squared difference terms in co-regularization based multi-view semisupervised learning (SSL) approaches [Krishnapuram et al., 2005; Sindhwani and Rosenberg, 2008; Sindhwani et al., 2005; Farquhar et al., 2005; Belkin et al., 2006], which are also regarded as the "co-training" approaches [Blum and Mitchell, 1998; Zhou and Li, 2010]. However, different from the SSL setting which focuses on better utilizing the additional unlabeled training data, the scenario of classification with hidden information assumes that hidden information is only available during training but not available during testing. We want to use such information to improve the target object classifier built on primary features. Moreover, the squared difference term in co-regularization SSL approaches are imposed on the unlabeled samples while it is imposed on labeled samples in our proposed approach.

The proposed method can be applied to different types of linear classifiers such as logistic regression (LR) and support vector machine (SVM) by selecting different loss functions. For instance, if we use hinge loss as l(yi↪ xi;w) =

max(0↪ 1 - yiw>xi), the objective function in Equation 2 would apply our additional information modeling to the SVM learning. Since the hinge loss is still convex, subgradient based optimization can be used to solve the objective. In this paper, we apply the it to the LR model. The loss function of LRmodel is:

where yi 2 ¶-1↪ 1♦. Since the LR loss function term is convex and differentiable, gradient based methods can be applied to solve the objective function.

For linear models, the dissimilarity regularization term in Equation 2 can be written as:

where X = [x1↪ x2↪ ▷ ▷ ▷ ↪ xN]> denotes the training data matrix, A = [a1↪ a2↪ ▷ ▷ ▷ ↪ aN]> denotes the hidden information matrix, C = [X↪ -A]>[X↪ -A], and w = [w>↪ w⇤>]>.

Since w>Cw = ¶[X↪-A]w♦T ¶[X↪-A]w♦ ≥ 0, the matrix C is positive semi-definite for any vector w. Thus, the score similarity term is also a convex quadratic term. Its gradient with respect to vector w is:

This gradient can be directly combined with the gradients of the remaining terms in Equation 2 to optimize the objective for the learning of parameters w and w⇤ .

3.2 Object-Attribute Relationships as Hidden Information

To further improve the performance of object recognition with hidden attributes, we propose to exploit the additional information in the attributes, i.e. the relationships between objects and attributes. As a set of semantic descriptions about the objects, attributes hold strong relationships with categories of objects that are determined by the intrinsic properties of different categories of objects. For instance, the object "bird" holds co-occurrence relationship with attribute "has wing", and holds mutually exclusive relationship with attribute "has horn". We believe such relationships, if captured as additional hidden information, would enforce the object classification to fit not only with the object labels, but also with the intrinsic properties of objects. In this way, the classifiers learned with attributes as hidden information can generalize better in the testing data.

To simplify the analysis, we consider the relationship between object label y and each of the M types of attributes, i.e. am with m 2 [1↪M], in a pairwise manner. Suppose the relationship between object label y and attribute am can be evaluated by a d dimensional real valued vector tym. Also, the relationship between predicted object ˆ y and attribute am can be evaluated by another d dimensional real valued vector ˆ tym. Since the predicted object ˆ y is given by the mapping function f : X 7-! R with parameter w, ˆ tym should also be a function of w as ˆ tym(w).

The essence of our method here is enforcing the relationship between the predicted object ˆ y and each of the attribute label to be close to the relationship between the ground truth object label and the corresponding attribute label. In this way, our classifier learning can be connected with the hidden information. Such a regularization is natural, since a perfect object classification should also preserve the relationships between objects and attributes perfectly. Suppose the above two relationships can be evaluated by vectors ˆ tym(w) and tym respectively, we can hence enforce the `2 norm of the vector difference ˆ tym(w) - tym to be small. Combining such a relationship regularization with the terms for standard object classifier learning as in Equation 1, our general formulation for exploiting object-attribute relationships can then be written as in Equation 5.

Compared to the formulation in Equation 2, this formulation does not require to learn a hypothetic classifier and is therefore more computationally efficient.

To fulfill the general formulation in Equation 5, we further introduce the detailed definition of ˆ tym(w) and tym. We utilize the linear regression coefficients to evaluate the relationships between object y and each of the attribute am. Here, the regression coefficients rym and sym reconstruct the object label y from the attribute am as y = rym + symam. Coefficients rym and sym can then be obtained by minimizing the mean square error as:

where ami is the value of attribute am for sample i.

Both the coefficients rym and sym have their specific meanings for representing the relationship. When y and am are binary values with "1" standing for positive label and "-1" standing for negative label, sym will reflect the "cooccurrence" (sym > 0) and "mutually exclusive" (sym < 0) relationships with its amplitude indicating the extent of the relationship. When sym ⇡ 0, the two variables tend to be "unrelated". Also, rym represents the bias between y and am values. It gives the prior information on whether y would be more frequent to present than am or not.

Here, we define the matrix φm, the object label vector y, and the relationship evaluation vector tym as:

Vector tym can then have a closed form solution as tym = (φ>mφm)-1φ>my ⌘ φ+my, where φ+m is the Moore-Penrose pseudo-inverse [Penrose, 1955] of matrix φm. Given the object and attribute labels in the training data, vector tym should be a constant unrelated to classifier parameter w.

The predicted object ˆ y can be further represented by the object classifier response w>x. Hence, the regression coefficients ˆ rym and ˆ sym should reconstruct w>x from attribute am as w>x = ˆ rym + ˆ symam. These two coefficients are obtained by minimizing the following mean square error:

Define the training sample matrix X, and the relationship evaluation vector ˆ tym(w) as:

then the closed form solution for relationship evaluation vector is ˆ tym(w) = φ+mXw.

Now, we replace the ˆ tym(w) term in Equation 5 as φ+mXw, and keep the pre-calculated constant term tym. The complete objective function formulation can then be given in Equation 6.

From Equation 6, we can see our formulation capturing object-attribute relationship would bring in an additional quadratic term w>X>(φ+m)>φ+mXw. Reshaping this term, we find it equals to (φ+mXw)> (φ+mXw) ≥ 0 for any w. Hence, the matrix X>(φ+m)>φ+mX is positive semi-definite, and the whole quadratic term is then convex. Such a quadratic convex term is easy to optimize.

Similar to our previous formulation discussed in Section 3.1, the formulation in Equation 6 can also be applied to different types of linear classifiers such as logistic regression (LR) and support vector machine (SVM) by selecting different loss functions. As discussed in Section 3.1, if we use hinge loss as l(yi↪ xi;w) = max(0↪ 1 - yiw>xi), the objective function in Equation 6 would apply our relationship modeling to the SVM learning. Since the hinge loss is still convex, subgradient based optimization can be used to solve the objective. In this paper, we apply the relationship modeling to the LR model. The optimization of the LR loss functions have been discussed in Section 3.1.

For our formulation in Equation 6, the gradient of the relationship term can be represented as:

Since the relationship term in Equation 6 is still convex, we can directly combine the gradient in Equation 7 with the gradients of LR loss functions and the `2 norm parameter regularization term for object classifier learning.

3.3 Combined Formulation

The formulation discussed in Section 3.1 utilizes hidden attributes as additional information, and enforce the score dissimilarity between the hypothetic model on the hidden attributes and the target model on the image feature to be small. On the other hand, the formulation in Section 3.2 models the relationships between hidden attributes and the object. It enforces preservation of the relationships between attributes and the object category. These two formulations incorporate different properties of hidden attributes, and hence can be further combined into one objective function. The combined objective function can be written as:

This objective function hence minimizes the score dissimilarity term and the relationship regularization term simultaneously during the learning of parameters w and w⇤ .

The gradient of the objective function in Equation 8 with respect to w and w⇤ is the combination of the gradients for each term in this equation. The gradients of our proposed score dissimilarity term and the relationship regularization term are given in Equation 4 and Equation 7 respectively. Since each term in Equation 8 is convex, the optimization can be solved by gradient descent based methods.

4 Experiments

We perform experiments on natural scene object classification on two benchmark datasets: aPascal dataset [Farhadi et al., 2009] and Animals with Attributes (AWA) dataset [Lampert et al., 2009]. The goal is to compare the performance of our proposed approaches incorporating hidden attributes with the basic approaches without using attributes, the traditional attribute approaches using attributes as the middle level representation, and the existing approaches for learning with hidden information.

Models. The models evaluated in our experiments include: the standard logistic regression model (LR) and support vector machine (SVM) learned with only the training data, the proposed formulation in Equation 2 using attributes as extra information (xLR+), the proposed formulation in Equation 6 with incorporating attribute relationships (LR-Rel+), the formulation in Equation 8 further combining formulations in Equation 2 and Equation 6 (xLR-Rel+).

The aPascal dataset contains 6340 training images and 6355 testing images collected from Pascal VOC 2008 challenge. Each sample belongs to one of the twenty object categories: people, bird, cat, cow, dog, horse, sheep, aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa, and tv/monitor. The dataset also provides 9751 dimensional base feature for each of the training and testing sample. Various types of color, textural, HOG, shape and edge descriptors are combined with a Bagof-Words approach to formalize the 9751 dimensional base feature vector. The base feature is used in all of the following experiments.

A list of 64 attributes are annotated for each sample in the dataset with examples shown in Figure 1. Each attribute is quantized into "-1" or "1" binary values to represent the absence or presence of the attribute. These hidden attributes are used for the proposed algorithms xLR+, LR-Rel+ and xLRRel+. We use a one-versus-all strategy to perform multi-class object classification in this dataset. A total of 20 classifiers are trained to predict each category against the remaining categories. The final decision is made by comparing the scores for each object type.

During classifier learning, the coefficients are tuned through a two fold cross validation procedure within the training set. The results are shown in Table 1, where both the overall accuracy rate and mean per-class recognition accuracy rate are given. We also include the results from state of art middle level representation work [Farhadi et al., 2009; Wang and Mori, 2010; Wang and Ji, 2013], and the results from the state of art learning with hidden information approaches include the SVM+ approach [Vapnik and Vashist, 2009] and the LR+ approach [Wang et al., 2014] in Table 1.

Firstly, we compare the baseline LR approach with our proposed models xLR+, LR-Rel+, and xLR-Rel+. From the results, we can see that by incorporating hidden attributes, the proposed models xLR+, LR-Rel+, and xLR-Rel+ all outperform LR in terms of both the overall and mean per-class accuracies, which shows the effectiveness of the proposed algorithms. In addition, xLR-Rel+ outperforms both xLR+ and LR-Rel+ in both accuracy evaluations. This comparison shows that the combination further improves the performance.

Secondly, we compare with state of art middle level representation approaches by [Farhadi et al., 2009], [Wang and Mori, 2010] and [Wang and Ji, 2013]. We can see that although predicted attributes are not utilized during testing, all three of our models (i.e. xLR+, LR-Rel+ and xLR-Rel+) outperform the approach proposed in [Farhadi et al., 2009]. Compared to the results by [Wang and Mori, 2010], our xLRRel+ approach performs better in overall recognition rate by around 4%, and performs lower in mean per-class recognition rate by about 3%. This is expected. As argued in [Wang and Ji, 2013], [Wang and Mori, 2010] use the loss function specifically designed for skewed data, and the aPascal data is skewed by having 2571 of 6355 testing samples to be in "person" category. [Wang and Mori, 2010] also report their performances with the standard "0/1" loss function. Results are 46.25% for mean accuracy, and 62.16% for overall accuracy, which are both not as good as our performances. The approach in [Wang and Ji, 2013] also combines the attribute relationship in the model, and its object recognition performance in aPascal is not as good as our xLR-Rel+ model for both overall and mean per-class evaluations. These results show that our approaches are quite effective for improving object classifier learning compared to traditional middle level representation methods.

Thirdly, we compare our methods with learning with hidden information approaches including the SVM+ approach [Vapnik and Vashist, 2009] and the LR+ approach [Wang et al., 2014]. From Table 1, we can see all our three models (i.e. xLR+, LR-Rel+ and xLR-Rel+) outperforms both SVM+ and LR+ approaches in both the mean and overall recognition accuracy. We perform the Wilcoxon rank sum test to evaluate the performance improvement of the proposed xLR-Rel+ model over both LR and SVM+ approaches. Both tests show performance improvements are statistically significant with a p-value less than 5%.

To further compare the performances of our proposed model with the learning with hidden information state of the arts including SVM+ and Rank Transfer, we test the proposed algorithm for object classification on the Animals with Attributes (AWA) dataset [Lampert et al., 2009]. This dataset includes 6180 images that belong to 10 testing classes. These 10 testing classes are different wild animals including chimpanzee (CP), giant panda (GP), leopard (LP), persian cat (PC), pig (PG), hippopotamus (HP), humpback whale (HW), raccoon (RC), rat (RT), and seal (SL).

To compare with the results in [Sharmanska et al., 2013], we follow the same experimental setting as in [Sharmanska et al., 2013]. In such setting, the models are tested on recognizing each possible pair of the 10 animal classes. This would give us 45 animal pairs. Also, the provided SURF descriptor in 2000 dimensions are used as features, and the predicted attributes in the format of probability estimates provided by [Lampert et al., 2009] are used as hidden information during training. With the provided feature and hidden information, 45 binary object classifiers are trained for each animal pair. We use 100 samples per object class for training, and 200 samples per object class for testing. As in [Sharmanska et al., 2013], we repeat such training/testing split procedure for 20 times. The results with comparisons to SVM+ and Rank Transfer methods are presented in Table 2

From the results in Table 2, our proposed xLR-Rel+ is very effective for incorporating hidden attributes. Among the 45 possible cases, the SVM performs the best only in 1 case, the SVM+ performs the best in 7 cases, the Rank Transfer method performs the best in 11 cases, and our proposed xLRRel+ model performs the best in 26 out of the 45 cases. The average values of accuracies over the total 45 pairs also show that our proposed model performs better than SVM, SVM+, and Rank Transfer methods.

5 Conclusion

In this work, we propose to incorporate hidden attributes for object classification. Instead of predicting these attributes explicitly or implicitly during testing, we utilize the attributes only during training to improve the learning of the object classifier on the primary features. We develop two different approaches to incorporate the hidden attributes, with one approach utilizing attributes as additional information, and the other incorporating the relationship between attributes and objects. Finally, these two different approaches are combined into one learning objective. We evaluate our approach on the natural scene object classification. Experiments demonstrate the effectiveness of our approaches for classification over state of the art methods on benchmark datasets.

Acknowledgments

This work is funded in part by US Defense Advanced Research Projects Agency under grants HR0011-08-C-0135-S8 and HR0011-10-C-0112, and by the Army Research Office under grant W911NF-13-1-0395.