Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data

1 Introduction

Confusion has been found to hinder user experience and satisfaction with user interfaces [e.g., Nadkarni and Gupta 2007] and Information Visualization (InfoVis) [e.g., Lee et al. 2016; Yi 2008]. To date, most work on developing methods to detect and prevent or resolve confusion in realtime during interaction has been in the field of Intelligent Tutoring Systems (ITS), [Bosch et al. 2015; Baker et al. 2012; D'Mello and Graesser 2007]. Building on this work, we investigate how to predict confusion during the interaction with a visualization-based interface, with the long-term goal of devising intelligent user-adaptive visualizations that can provide personalized interventions to help confused users. Such user-adaptive visualizations would be especially beneficial as complex visualizations are becoming increasingly used by broad audiences, not only in professional settings, but also for personal usage (e.g., for monitoring health and fitness, interactions in social media, and home resources consumption) [Huang et al. 2015].

In this paper, we investigate machine learning models to predict occurrences of confusion during the interaction with ValueChart, an interactive visualization to support multicriteria preferential choice. To make such predictions in real-time, we leverage both interaction data as well as eye tracking capturing users' gaze patterns, pupil size and head distance to the screen. Interaction data have been used before to predict confusion in computer games [Pentel 2015] and ITS [Baker et al. 2012]. However, we are the first to study how eye tracking can be used to predict confusion, with the rationale that eye tracking should be particularly informative in predicting confusion during visualization processing, as visual scanning is a fundamental component of working with a visualization. Furthermore, eye tracking has been shown to be a good predictor of affective states in educational systems [Jaques et al. 2014; Muldner et al. 2010] as well as user characteristics in InfoVis, [e.g., Steichen et al. 2014; Jang et al. 2014].

This work has two main contributions. The first is a proof of concept that confusion can be predicted in real-time in InfoVis, with 61% accuracy (significantly better than chance) for occurrences of confusion and a false positive rate of only 7.4%. The second contribution is evidence of the importance of eye tracking for building predictors of confusion in InfoVis, with the most informative sources of information being differences in the user's attention to the labels of the InfoVis, as well as variations in user pupil size and head distance to the screen.

2 Related Work

Previous work suggests that confusion can negatively impact user experience or satisfaction with an interface, e.g., [Rickenberg and Reeves 2000; Nadkarni and Gupta 2007]. In InfoVis, confusion has been linked to lower user performance and satisfaction in completing decision making tasks [Yi 2008], and was shown to affect novice users when they process an unfamiliar visualization [Lee et al. 2016].

In the field of Intelligent Tutoring Systems, predictors of confusion have been built by leveraging facial expressions [Bosch et al. 2015; D'Mello and Graesser 2007], posture [D'Mello and Graesser 2007], or students' interface actions and studying behavior [Baker et al. 2012]. In HCI, Pentel [Pentel 2015] leveraged mouse usage information to predict occurrences of confusion in a simple computer game. We extend this work by showing the feasibility of predicting confusion in real time in InfoVis, and we extend previous work on confusion prediction in general by using a new sensor, eye tracking.

Eye tracking has been shown to be a good predictor of other emotional or attentional states such as mind wandering while reading [Bixler and D'Mello 2015], as well as boredom, curiosity and excitement, while learning with educational software [Jaques et al. 2014; Muldner et al. 2010]. In InfoVis, gaze and pupil data have been investigated to predict long-term user traits (e.g., perceptual speed, visual and verbal working memory), as well as shortterm properties such as task completion time, learning curve, or intention for visual search [Steichen et al. 2014; Lallé et al. 2015; Jang et al. 2014]. Still in InfoVis, [Yelizarov and Gamayunov 2014] tracked mouse and keyboard events to predict users' level of cognitive load and adjust the amount of information to be displayed accordingly. Their results showed that their adaptations positively impacted users' performance.

3 User Study

ValueCharts. Complex decisions can often be framed as preferential choices, i.e., the process of selecting the best option out of a set of alternatives characterized by a variety of attributes (e.g., select a car to buy, a university to attend, etc.). The dataset 1 used in this paper was collected from a user study using ValueChart, an interactive visualization for preferential choice [Conati et al. 2014]. Figure 1 shows an example of ValueChart for selecting rental properties among ten available alternatives (listed in the leftmost column), based on a set of relevant attributes (e.g., location, appliances, etc.). These attributes are arranged hierarchically in the top part of the ValueChart, with a column for each attribute in the central part of the display. The width of each column indicates the relative weight assigned to the corresponding attribute. The available alternatives (i.e., rental homes) are represented as the rows in the display. Each cell specifies how the alternative in that row fares with respect to the attribute in that column, indicated by the amount of filled cell color. In the rightmost part of the ValueChart, all values for each alternative are accumulated and presented as stacked bars, displaying the overall value of each alternative (e.g., home4 is the best home in the example in terms of overall value).

The interactive functionalities available to support the decision process include: (i) inspecting the specific domain value of each attribute (e.g., the rent of home1 being equal to $500), by left clicking on the related alternative; (ii) sorting the alternatives with respect to a specific attribute (by double-clicking on the attribute name); (iii) swapping attribute columns (initiated via a left click on one of the attributes); and (iv) resizing the width of an attribute's column to see how that would impact the decision outcome (initiated via a left click on the column edge).

ValueChart has been extensively evaluated for usability and adopted in several applications (e.g., [Yi 2008; Wongsuphasawat et al. 2012]). It has, however, inherent complexity due to the nature of the task, which can still generate confusion in some users [Yi 2008; Conati 2013].

Procedure. 136 participants (age range 16 to 40, 75 female) were recruited from various departments at our university to perform 5 different types of tasks with ValueChart (e.g., retrieve the cheapest home or select the best home based on the aggregation of price and size). After 10 min of training with ValueChart, each participant repeated each task type 8 times in a randomized fashion to account for within-user variability, for a total of 40 tasks. This results in a total of 5440 trials (136 users × 40 trials).

1 Data available at http://www.cs.ubc.ca/~lalles/IJCAI16.html

While performing the tasks, the user's gaze was tracked with a Tobii T120, a non-intrusive eye-tracker embedded in the study computer monitor. In order to avoid possible confounds on pupil size due to lighting changes, the study was administered in a windowless room with uniform lighting. To compensate for physiological differences in pupil size among users, pupil diameter baselines were collected for each user by having them stare at a blank screen for ten seconds at the beginning of the study.

Collecting data on user confusion. Collecting ground truth labels is one of the main challenges for building user models that can predict transient user states in an adaptive interface (e.g., [Porayska-Pomsta et al. 2013]), and a variety of methods have been proposed in the literature to address this challenge (see [Conati et al. 2013] for an overview).

After careful consideration of various options, we chose to have users self-report their confusion by clicking on a button labeled "I am confused" (see Fig. 1, top right). Users were instructed to use the button as follows (rephrased for brevity): "[you should click the confusion button] if you feel that you want to ask the experimenter a question about something; if you are confused about the interface; if you are confused about the wording of a question. …These are just a few examples, to show that confusion can occur in many unforeseeable ways, [which are all] OK reasons to click the confusion button." Participants were told that clicking on the button would have no effect on the interaction, it was just included for data collection. At the end of the study, each participant was shown replays of interaction segments centered around their reports of confusion, to verify that the report was intended and elicit the reason of the confusion. This collection method was evaluated via pilot studies before being deployed in the main experiment [Conati et al. 2013].

Confusion was reported in 112 trials (2% of all trials), with 80 users (59%) reporting confusion at least once during the study, with an average of 1.4 clicks (SD=1.9). There was never more than one click per trial. Reasons for confusion reported by participants include not understanding the tasks, perceived ambiguity in the textual or visual components of the interface, interactive functionalities not working as expected, alleged missing functionalities to solve the task.

To investigate the impact of confusion on users' performance, we ran an MANOVA with task accuracy and task completion time as the dependant variables, and the occurrence of confusion (2 levels: YES or NO) and types of task (5 levels) as factors. The MANOVA revealed a significant main effect of confusion on both accuracy (F1,2159=108, p < .001, η 2 p=.02) and completion time (F1,2159=125, p < .001, η 2 p=.02), with confusion trials having a lower accuracy and being longer than no-confusion trials. These results confirm the negative impact of confusion on performance reported in [Yi 2008] for a visualization based on a variation of ValueChart. It is notable that the impact exists even with the relatively low number of confusion reports in our dataset.

4 Predicting Confusion in ValueChart

We label trials as "confusion" when the user pressed the confusion button at least once during the trial, or "noconfusion" otherwise. Thus, predicting confusion in our dataset is a binary classification task: classify each trial as one in which the user might be experiencing confusion, or not. We compare a variety of features sets for this classification task (Section 4.2). In Section 4.1, we describe the datasets we generated to compute these feature sets.

4.1 Data Windows

To simulate the real-time prediction of confusion episodes, we use only users' data prior to the click on the confusion button. As there is no such click in no-confusion trials, we randomly generate a "pivot point" in each of those trials. In order to ascertain how much data leading up to an episode of confusion is needed to predict it, we built our feature sets (described below) by using two different windows of data: a short window captures data 5 seconds immediately before a confusion click 2 (or pivot point); a full window captures the whole episode of confusion by including data from a click (or pivot point) back to the beginning of the trial 3 . Full windows were on average 13.7s in length (SD=11.3s).

5.2 Predictive features

The eye tracking data collected during the study provide information on user gaze patterns (Gaze, from now), on changes in a user's pupil width (Pupil), and on the distance of the user's head from the screen (Head Distance, defined as the averaged distance of each eye to the screen), which are all good candidates to explore as predictors of confusion. Confusion is likely to impact how a user attends to elements of the visualization (Gaze), which has already been successfully used to predict user performance and other abilities with visualizations [Steichen et al. 2014; Lallé et al. 2015; Nazemi et al. 2014]. Pupil size has been associated to cognitive load [e.g., Granholm and Steinhauer 2004], which might be affected when the user experiences confusion. Pupil has also been shown to be a predictor of other mental states, such as mind wandering [e.g., Bixler and D'Mello 2015]. Head distance provides a rough indication of user posture, which has been shown to be a predictor of user engagement with a task [D'Mello and Graesser 2007]. Since ValueChart is interactive, we also leverage data on mouse events, which have been shown to predict user confusion during interaction with a computer game [Pentel 2015]. From all these measures (Gaze, Pupil, Head Distance, and Mouse Events) we derive four groups of features (listed in Table 1) that we evaluate in terms of their ability to predict user confusion with ValueChart.

2 Windows actually end 1 second before a confusion click to avoid confounds associated with the specific intent to report confusion (e.g., fast straight saccades toward the button).

3 Note that this window cannot include another confusion click because there is only one per trial in our dataset. Otherwise, full windows would go back only to the last episode of confusion.

Gaze features (Table 1a) describe user's gaze patterns in terms of fixations (gaze maintained at one point on the screen), and saccades (quick movement of gaze from one fixation point to another). From this raw gaze data, we generated features that capture overall gaze activity on the screen, as well as activity over specific Areas of Interest (AOI), shown in Figure 2. We selected these features because they have been extensively used in HCI to capture differences in users' attention patterns over an interface [e.g., Holmqvist et al. 2015].

Pupil and Head Distance features (Table 1b) are generated by averaging the corresponding measures for each eye. Pupil size is adjusted using the pupil baseline collected during the study to get the percentage change in pupil size (PCPS), as defined in [Iqbal et al. 2005]. Features like the mean, min, max and std.dev are standard ways to measure fluctuations in a measure of interest, and have been used with pupil size to reveal individual differences while users work with an interface [Holmqvist et al. 2015] and with head distance to predict user boredom [Jaques et al. 2014]. We also included pupil size & head distance of the first and last fixation in the data window (see Section 4.1) as a way to capture variations of the measures between the closest and farthest data-points to the confusion click in that window.

Mouse Event features (Table 1c) are built upon summary statistics on left and double mouse clicks (the only two with associated functionalities), measured both over the whole screen and for the 7 specific AOIs defined in Figure 2. Mouse click rate has been shown to be effective at predicting cognitive states in other interactive tasks [Lim et al. 2015]. The time to first click has been used for early prediction of user failure with an interface [Liu et al. 2010], thus it might also be relevant to predict confusion because confused users are likely to make more errors.

From all the features described above, we derive a total of 8 feature sets that we will compare to predict confusion:

• x One feature set for each of Gaze (G), Pupil (P), Head Distance (HD), and Mouse Events (ME) features.
• x All features together (ALL).
• x Only features derived from eye tracking data (G+P+HD), as these can be used in non-interactive InfoVis as well.
• x Head distance and Pupil (P+HD) as these features are entirely independent from the layout of the visualization.
• x Head Distance+MouseEvent (HD+ME), as the expensive eye tracker is not needed to collect these features (a webcam can reliably infer the distance to the screen).

We used each of the two data windows described in Section 4.1 to generate each of the 8 features sets above, for a total of 16 combinations.

4.3 Addressing data imbalance

Our dataset is highly imbalanced, with only 2% of confusion trials. Building meaningful classifiers on imbalanced dataset is challenging: a majority class classifier that always predicts no confusion would have a very high accuracy, but would be useless. Thus, to train our predictors of confusion, we balanced training data by using the wellknown SMOTE algorithm for data over-sampling [Chawla et al. 2002]. Specifically, using SMOTE we generated "synthetic" confusion trials based on k nearest neighbors in the minority class (we used the default value k=5). To do so, SMOTE randomly duplicates a confusion trial (ct) and modifies it by sampling for each feature a new value along the line between ct and its k nearest neighbors. Next, noconfusion trials are randomly discarded (down-sampled) until the dataset is balanced. In our study, we over-sampled confusion trials by 200% (i.e., number of confusion trials is doubled) and 500%. These percentages appeared to be among the best ones when applied to similar imbalanced datasets in [Chawla et al. 2002], where they also show that more over-sampling is pointless.

4.4 Machine learning set up

To build our classifiers, we use Random Forest tuned with 100 trees using the Caret package in R [Kuhn 2008]. We chose Random Forest because previous work has shown that this learning algorithm performed well on similar prediction tasks [e.g., Pentel 2015; Wu 2015]. The classifiers are trained for all combinations of features sets (8), data window lengths (2) and SMOTE configurations (2) for a total of 32 classifiers. These classifiers are trained and evaluated with a process of 20-runs-10-folds nested crossvalidation, which includes two levels (inner and outer) of cross-validation.

At the outer level the following process is repeated 20 times (runs) to strengthen the stability and reproducibility of the results. Data is randomly partitioned in 10 folds; in turn, each of the 10 folds is selected as a test set; the remaining 9 folds are SMOTE-balanced and used to train a classifier which is then tested on the test fold. The performance of each classifier is averaged across the outer test sets, and then again over the 20 runs. It should be noted that test sets at the outer level are not altered by SMOTE in any way.

We performed cross-validation over users, meaning that in each cross-validation fold, all trials of a given users are either in the training or in the test set. We kept in the folds a distribution of confusion data points similar to that in the whole dataset. As the outer test folds are strongly imbalanced, model performance is measured via sensitivity and specificity, two suitable measures when data are skewed [Kotsiantis et al. 2006], defined as follow:

• x Sensitivity (or true positive rate): proportion of confusion trials that are correctly identified as such. It indicates the ability of the model to predict occurrences of confusion.
• x Specificity (or true negative rate): proportion of noconfusion trials that are correctly identified as such. It indicates the ability of the model to avoid false positives (as 1 - Specificity is the proportion of false positives).

At the inner level of the nested cross-validation, Correlation Feature Selection [Kuhn 2008] is applied to remove highly correlated features. Next, the best decision threshold is selected for each Random forest classifier using ROC curves plotted on inner train data only. The threshold selected is the one that gives the best trade-off between sensitivity and (1 - specificity) by taking the closest point of the ROC curve to the point (0, 1) representing perfect classification [Fawcett 2006].

5 Results

We analyze the performance of the 32 classifiers defined in Section 4.4 by running a 3-way MANOVA with:

• x sensitivity and specificity of the classifiers as the two dependent variables,

• x feature set (8 levels), window length (2 levels) and SMOTE configuration (2 levels) as the factors.

The MANOVA reveals a main effect of both feature set (F8,67=18.65, p < .001, η 2 p=.59) and SMOTE configuration (F1,26=9.58, p = .005, η 2 p=.17). No main effect of data window length, nor any interaction effects were found. To investigate further the two main effects, we run univariate pairwise comparisons using the Holm-Bonferroni adjustment for family-wise error 4 .

Table 2: Effect of feature set on model performance, with G=Gaze, P=Pupil, HD=Head Distance, ME=Mouse Event.

Effects of Feature set. Table 2 summarizes the results of the pairwise comparisons by ordering feature sets according to their mean sensitivity and specificity over data windows and SMOTE configurations. Bold underlining indicates models for which there are no statistically significant differences. For example, for sensitivity, these is no difference between G+P+HD and HD+P, they are both better than P but worse than ALL.

Results in Table 2 show that combining various data sources works usually better than using a single data source, for both sensitivity and specificity. In particular, the best feature set for both performance measures includes all the features (ALL). This indicates that eye tracking features work well together to predict confusion, and that adding mouse events can lead to significantly better accuracy. It is interesting to note that G+P+HD (the second best model) is not significantly better than P+HD, meaning that adding Gaze data to Pupil and Head Distance features sets does not lead to a significant improvement.

Effects of SMOTE configuration. The pairwise comparisons show that:

• x Sensitivity is better with SMOTE-200% than with SMOTE-500%, (t(198)=3.19, p = .002, η 2 p=.15).
• x Specificity is better with SMOTE-500% than with SMOTE-200%, (t(198)=2.41, p = .017, η 2 p=.04).

These results show that there is a trade-off between SMOTE-200% and SMOTE-500% in terms of sensitivity and specificity. In particular, generating more synthetic data with SMOTE has a substantial negative effect on the sensitivity (see medium effect size η 2 p=.15). This is likely due to the fact that synthetic data start to dilute the information captured in the real confusion data points. This effect is inverted for specificity, although the effect size is small (η 2 p=.04), indicating limited implications in practice.

5.1 Model performance

In this section, we analyze in more detail the performance (in terms of sensitivity and specificity) of the classifiers using the ALL and P+HD feature sets and trained over "Full Window" 5 data. We focus on these two feature sets because ALL showed the best overall performance (Table 2) and P+HD is the second best model along with G+P+HD, but it has the advantage to be fully independent from the layout of the visualization. For each of these feature sets and Full window, we report results for both SMOTE-200% and SMOTE-500% (Table 3), since Section 5.1 showed that there is a trade-off between these two configurations.

4 Actual values for average sensitivity and specificity will be discussed later for specific models.

Overall, results in Table 3 indicate that SMOTE models have very similar specificities, with variations of only about .02 for both ALL and P+HD, whereas SMOTE-200 achieved by far the highest sensitivity 6 , reaching .61 for ALL. Thus SMOTE-200 used with all features together appears to be the most promising model to predict user confusion during interaction with ValueCharts. However, layout independent information captured only by pupil and head distance features appears to successfully predict 57% of confusion trials on unseen data, a very encouraging result in terms of building visualization-independent classifiers.

5.2 Feature importance

Our results show that multiple data sources together (i.e., ALL Features) can better predict confusion, than any single source. To ascertain which features best predict confusion, we use the method described in [Liaw and Wiener 2002] to measure feature importance in Random forest classifiers. Table 4 shows the top 10 selected features for our best performing classifier (using the ALL feature set and SMOTE -200%), with relative importance normalized between 0 and 100. In the table, a positive (negative) direction of the effect column (D) indicates that the value of the feature is higher (lower) in confusion than in no-confusion trials.

Pupil features. Table 4 shows that overall pupil size features are the most important predictors of confusion, as the top three best features belong to this set. Generally, increase in pupil size is correlated with higher cognitive load [Granholm and Steinhauer 2004]. Consistently with this result, end, max, and mean pupil size are higher in confusion trials, where confused users might be experiencing a higher cognitive load. Also, users in the confusion trails have higher std.dev pupil size, which might be an indication of a higher variability in their cognitive load compared to more stable non-confused users. The prominence of pupil-based features as predictors of confusion is especially promising for the generalization of our work, since these features are independent from the layout of the visualization, and thus may be used to predict confusion in other InfoVis.

5 Windows length had no effect on performance, so either length could be used here.

6 When differences in specificity is higher, measures such as the practical utility could be used to investigate the trade-off between sensitivity and specificity [Gena 2005].

Head distance. There is one head distance feature among the top 10, namely std.dev head distance. Confused users have a higher value for this feature, suggesting that confusion generates more fluctuations in the user's position. Distance to the screen has been shown to be correlated with user engagement [D'Mello and Graesser 2007], thus more fluctuations in the position of confused users might indicate that these users get closer to the screen to better attend to the unclear information before eventually disengaging. As with pupil size, head distance to the screen does not depend on the visualization layout and thus may be a good predictor of confusion using other InfoVis. Also as noted before, head distance may be inferred with a cheaper webcam.

Gaze features. Five of the 10 most important features in Table 4 are Gaze features related to attention to the AOI that includes the name of the attributes (Labels_attr) in the decision problem (see Fig. 2). These features are time to last fixation in the AOI, longest fixation, proportion of time spent, as well as transitions from and to the AOI that encloses the name of the problem alternatives (Labels alter). Confused users have higher values for all these features, suggesting that they need to process the names of the attributes more extensively. One possible explanation for this trend is that labels are a source of confusion (e.g., due to ambiguous wording). Another is that looking at names of attributes and alternatives is a way to reduce confusion as text in visualization is meant to support data comprehension. It should be noticed that all Gaze features here relate to a particular AOI, thus are layout dependent. This indicates that although many independent features (e.g., pupil and head distance) are important predictors of confusion, additional information about specific components of the visualizations can improve prediction.

Conclusion

If confusion could be predicted and resolved in real time, user experience and satisfaction with InfoVis would be greatly improved. In this paper, we focused on predicting occurrences of confusion during the interaction with ValueChart, an interactive visualization to support multicriteria preferential choice. To this end, we leveraged a user study that collected ground truth labels for confusion, along with eye tacking and interaction data. Then, we compared various combinations of these data sources to train Random forest classifiers for confusion, and technically had to deal with data imbalance.

Our results show that eye tracking is valuable to predict confusion in real time. Remarkably, we found that 61% of the occurrences of confusion can be predicted, while getting a false positive rate of only 7.4%. More tellingly, when we examine the most important features used by the classifiers, it appears that our models are able to capture aspects of the interaction that are very plausibly related to confusion. Furthermore, some of these features may well generalize to other visualizations. For instance, we found that features of pupil size (which are independent of the layout of the current visualization) are strong predictors of confusion, consistently with the fact that pupil size is correlated to cognitive load, which plausibly correlates with confusion. Another strong visualization-independent predictor was a feature related to head distance. Again this makes sense, because confusion can affect engagement with a task, which has been shown to be predictable by head distance. Additionally, prominent gaze features reveal differences in user's attention to the labels of the visualization among confused and non-confused users; however, these features may not generalize so easily to other visualizations.

To increase the performance of our models, future work includes improving our features and models selection via ensemble modeling, leveraging additional features such as facial expressions, and using past observed data to optimize prediction for each individual user. Another thread of future work relates to further investigating the generalizability of our findings to other visualizations, as well as researching how confusion can be addressed once predicted.

Acknowledgement

This research was funded by NSERC Grant 227795-10. We thank Enamul Hoque and Dereck Toker for their help with data collection, and Dereck for his help in revising the paper.