Fine-tuning Deep Neural Networks by Interactively Refining the 2D Latent Space of Ambiguous Images

1 Introduction

At present, Deep neural networks (DNNs) maintain excellent results in classification. However, they are still difficult to extract enough features to distinguish the ambiguous data that are very similar but belong to different classes. Compared with the networks, humans may have better classification capabilities for a small number of pictures or a group of abstract pictures. For example, as shown in Figure 1, in a large sketch dataset called Quick Draw [Jongejan et al., 2016], many abstract hand-drawn pictures are difficult for machines to distinguish, while humans can distinguish them well.

Human-in-the-loop is a promising direction to leverage human knowledge to help machines solve intractability problems in the field of artificial intelligence [Nashed and Biswas, 2018], many works apply this idea [Igarashi et al., 2016; Huang and Canny, 2019]. Moreover, a lot of works now revolve around interactive interfaces [Larsson et al., 2020; Wang et al., 2014], where users make changes to the data on the workspace to help machines do tasks that are difficult for them to do. [Yang et al., 2019] proposed an approach that eliminates the problem of co-occurrence bias in the identification process through human participation. [Sakata et al., 2019] introduced a new network called CROWNN that engages people in the classification process, learns how humans classify, and then better implements the classification task. [Asai et al., 2020] combine a code editor with a scatterplot editor so that users can visually observe the data and modify the data interactively. [Chang et al., 2021] focus on labeling tasks, aiming to improve the efficiency and accuracy of human labeling.

In this paper, we describe a novel interactive approach to improving network accuracy by refining the 2D latent space of the inputs 1 . We first give a general overview of our work, introduce the rationale of our method, and then describe our experimental procedures and illustrate our conclusions with examples. At the same time, we designed an interactive user interface to validate our idea. We project high-dimensional features in a neural network onto a twodimensional workspace, the position information of the projected point reflects the classification result of the point to a certain extent. Users can move the inputs they think are classified as wrong to the positions they think are correctly classified. Then, through the interactive user interface, we designed, the results of the movement can be fed back to the neural network for re-training, to achieve the purpose of finetuning the neural network. We conduct user studies to evaluate the performance of our proposed method, the feedback from users shows our method obtained the expected results.

Our method has the following benefits: 1. Help the network to distinguish ambiguous data and achieve the purpose of improving classification accuracy. 2. Get a more understandable latent space for the user. Through the user interface, we designed, the positional relationship and distance between classes can be adjusted. 3. Guide the learning process of networks. Not only does it allow the network to learn features faster, but also allows small networks to achieve the same level of performance as large networks.

1 https://youtu.be/5EvDa3XiTiY

2 Method

We add a projection branch to the pre-trained network to allow direct and convenient use and modification of the latent space on the screen, and we use dimensionality reduction methods to project higher dimensions into 2D. Here we use principal component analysis (PCA) as a dimensionality reduction method. First, the high-dimensional vectors are converted into two-dimensional vectors through PCA, and then the result obtained by PCA is used as the supervision value to train the projection branch. After the network training is completed, the data can be projected onto our two-dimensional workspace through the projection branch, realizing data visualization. At the same time, the user can modify the coordinate values of the projected points in the workspace to fine-tune the network, as shown in Figure 2.

Fine-tuning Loss Function. The result of DNN classification is controlled by the loss function. The loss function is weighted by the classification loss and the distance difference loss before and after adjustment (see Formula 1). The classification loss losscls is obtained by performing cross-entropy calculation (CE) between the predicted label and the true label. The distance difference loss lossdis can only be obtained from these points whose positions have been adjusted. The specific method is to calculate the difference between the coordinates before and after the movement through the L2 norm. The parameters (wcls, wdis) are used to adjust the balance between the two losses. Through these losses, the network can be fine-tuned with the points the user moves. And then, the output result will be closer to the result changed by the user, thereby achieving the purpose that the user can assist the network in better classification.

User interface. We design a user interface to help users modify the 2D latent space, as shown in Figure 3. Our UI is divided into three parts from left to right, namely the function bar, the modification window, and the result window. The reset button in the function bar is used to restore the two windows to the initial state, and the update button is used to transfer the modified content in the modified window to the network for retraining and updating the coordinate values in the result window. The coordinate values in the modification window can be changed, and the coordinates before and after modification are used to fine-tune the network. The result window is used to visualize the updated 2D coordinate values. We provide three methods for moving coordinate values, namely, dragging with the mouse to move a single coordinate, using the lasso tool or the selection box tool to move multiple coordinates.

3 Experiments

Experimental Data Selection. The reasons for the emergence of ambiguous data can be divided into two parts. One is that there are deviations in the sketch data itself, which is caused by human factors such as the sketches drawn by the painters not being clear enough or not being careful enough. Second, there are different degrees of similarity between sketches of different categories, which will hinder the effect of network classification. With the improvement of the similarity between the sketches, the difficulty of network classification will also increase. Our experiments revolve around ambiguous data. We use the Quick Draw dataset to test our method. It is a large sketch dataset with over a billion hand-drawn sketches. This dataset provides images of various kinds and formats. We selected nine kinds of sketch data for experiments and used the DNN to complete the image classification task. The nine types of images selected are shown in Figure 1 B), these images have a certain degree of similarity, and people will easily distinguish them.

Training Details. We performed all our experiments using PyTorch. The base network we use for our experiments consists of four convolutional layers and three fully connected layers. The training and fine-tuning process of the network is optimized using the stochastic gradient descent (SGD) (with momentum). We train the training set until the training result is stable, and obtain the initial pre-training model. During training and fine-tuning, the learning rate and momentum are set to 0.04 and 0.9, respectively. All fine-tuned results go through one to three epochs. We set (wcls, wdis) to be (0.89, 0.11) respectively. The number of each type of test data is 500, and the number of points that we move in the experiment ranges from 1 to 30.

Experimental Procedure and Results Analysis. We selected two kinds of ambiguous data for experiments. The data of these cases has the characteristics that it is difficult for machines to distinguish them correctly, but it is easy for humans to distinguish them. We test the data of these cases separately, as shown in Figures 4 A) and B). The test set accuracy rate before moving is 86.33%, and when the network is finetuned with proper movement, the accuracy can be achieved at 86.48% and 86.36%, respectively. It can be seen from the test results that the correct motion can improve the accuracy of network classification, and the location of the updated data also moves closer to the location where the user moves, making the location more reasonable. At the same time, the correct motion can also adjust the predicted value of the data with the wrong predicted value to the correct predicted value, as shown in Figure 4 B), the predicted value of the original data is green (representing the clock), and the predicted value of the shifted data turns red (representing the apple), completing the transition from the wrong predicted value to the correct predicted value.

We invited four users to evaluate our user interface. We give each user four to six minutes to understand and learn to use our user interface. Next, we assigned participants several tasks to determine the utility of our system. Tasks include validating whether our method improves accuracy and whether our designed user interface is convenient. All users agreed that our user interface is not only convenient to use but also the form of two-dimensional point coordinates that can intuitively express the network's classification of different sketch data. In addition, the method that can affect the effect of network classification by moving the coordinates of the sketch is very novel and has good development potential.

4 Limitations and Conclusion

The effect of our proposed method has been validated on small datasets, however, it should be further evaluated on the common large datasets. Besides, the projection method and distance matching between the 2D space and the original latent space requires further research.

In this paper, we propose an interactive user interface for improving the accuracy of network classification. By projecting the features obtained by neural network classification onto a two-dimensional workspace, users can manually change the positions of these projected two-dimensional points, eliminating possible deviations in the dataset, and then positively affect the results of network classification. Our work proves that the combination of neural networks and human-in-the-loop methods can transfer the advantages of humans to the network. Human participation can eliminate some of the deviations that may occur in the classification process of the machine and can improve the accuracy of neural networks in classification tasks. In future work, we will further work based on the above limitations.