Beyond Labels: Knowledge Elicitation using Deep Metric Learning and Psychometric Testing

1 Introduction

Supervised learning with a flat set of labels is the panacea of machine learning (ML) application to a wide range of problems in many fields. In practice, this can be an excitingly limiting the success of the application. One can imagine a scenario of a medical diagnosis involving images (for example diabetic retinopathy) and a set of diagnostic images (e.g. photography of the eye retina) and a set of labels denoting the severity of the disease (e.g. 'no disease', 'severity 1', ... 'severity 4'). From the very beginning, the domain expert is forced to project her comprehensive knowledge on this flat single axis. This, in turn, limits our model both from its performance as it cannot learn from the full depth of knowledge as well from its interpretability as projecting to a fixed set of points further contributes to the 'black-box-ness' of our model.

In contrast, adding a more rich hierarchical structure to the data adds a significantly larger understanding of the finer subtleties of the data stemming from the relationships between the concepts in the domain.

The reason why so many solutions are formulated as mapping of datapoints (commonly high dimensional) to a fixed set of labels is that eliciting the knowledge from the domain and forming a training dataset is a difficult task. A data point can have a rich set of properties associated with it and eliciting an expert to provide an exhaustive annotation for each datapoint is prohibitively unscalable.

In our thesis, we propose an approach which able to elicit individual knowledge in efficient, easy to carry way. Psychometric test [Gescheider, 2013] is introduced to capture the user's latent perception. It is to assess a participant's ability to perceive differences between different objects, which has been adapted to measure complex images and videos [Menkovski and Liotta, 2012; Feng et al., 2014; Hadizadeh and Baji´c, 2013]. The captured differences is mapped to an embedding space by metric learning, where the em-bedded data are then structured in a hierarchical fashion that allows assigning properties and annotations to a large number of data points in a scalable way.

2 FinishedWork

We already tested our work in image data-sets. To elicitate hierarchical knowledge from image data sets, generally, there are two approaches. One is labeling each data point manually with multiple tags, then building a hierarchical structure with them. However, without pre-defined labels, annotators could give labels arbitrarily, and that could lead to a disorder and meaningless hierarchic structure. Even if we have the pre-provided labels, it is still a quite time-consuming and laborious method. On the other hand, some scholars try to group hierarchical images automatically based on language or visual features correlations. A common way to do these is to take advantage of the existing word semantic information (e.g.,WordNet) or the extracted images feathers. While this method is more efficient, it has some limitations: (1) for some data sets there is no prior word semantic information provided (e.g., in some expert domain) (2) The word semantic information or extracted images feathers can not represent user's perception, so the formed construct may not catering to the user's individual needs.

To cope with these limitations, we proposed an efficient hierarchical knowledge elicitation (HKE) framework as shown in is Fig. 1.

One of the main differences between our work and others is the way to extract the latent hierarchical knowledge from annotators' mind. Capturing it directly could be difficult, because annotators need the know the label classes in advance, and for one object the annotator has to give multiple hierarchical labels. Here, we take advantage of the psychometric discrimination test [Gescheider, 2013] to simplify the process. To be specific, a three-alternative-forced-choice test (3AFC) [Green et al., 1966] is employed in our work. As shown in Fig 3, the annotators are given 3 objects a1, a2, a3, and are forced to choose the most dissimilar one among them. By carrying out this simple test, we can elicit the extract hierarchical knowledge from the data by simple ternary decision.

Psychometric testing offers many advantages. However, the number of all possible discriminative tests is k combination of n, where k is 3, and n is the number of datapoints in the dataset. Asking all possible questions is typically not

Question: which object is the most dissimilar to the other two objects?

feasible. To address this, we develop a question selection approach using a Bayesian-based active learning method that selects questions with high uncertainty and high utility.

Furthermore, We propose our dual-triplet loss with adaptive margin to construct a more clear hierarchical structure.

We provide an empirical evaluation on a synthetic data set of simple shapes. The results demonstrate that our method is able to uncover hierarchical structure on image data sets, and confirm the shape bias in the responses of a human participant as shown in Fig. 2.

3 Limits and FutureWork

(1) we only studied single participator now, we are also interested in whether the model can elicit specified hierarchies when there are multiple users who do not fully agree with each other, and if there is a general hierarchic structure that captures the shared perception of all the users. (2) It would also be interesting to see other stimuli such as voice be researched using such a method. Classification of someone's voice whether they are angry or happy can also be a difficult task if we work with labels since they are not always very obvious. Therefore, using this method will potentially embed voice recordings in such a way that they become more interpretable. This can be very useful to any company having to deal with customers via the phone and able to detect quickly what the mood of someone is.