Item Response Ranking for Cognitive Diagnosis

1 Introduction

Recently, online education systems has been widely used [Jiang et al., 2019; Bi et al., 2020; Liu et al., 2019b]. These systems provide a variety of applications which can not only assist tutors to give proper instruction based on individual characteristics, e.g., strengths and weaknesses, of students, but also help students be aware of their learning progress [Pardos and Heffernan, 2010]. One of the key fundamental technologies supporting these systems is cognitive diagnosis, which tries to profile students by discovering their latent cognitive proficiency on knowledge concepts.

Massive efforts have been undertaken to improve the solutions of cognitive modelling. Generally, in cognitive diagnosis models (CDMs), students are characterized by their proficiency in specific knowledge concepts (e.g., Circle Graph, Equivalent Fractions). Classic CDMs (e.g., Item Response Theory (IRT) [Lord, 1952; Lord, 1980; Rasch, 1961] and Deterministic Inputs, Noisy "And" gate model (DINA) [De La Torre, 2009]) use single-dimension or discrete variables to represent the latent trait features of students and test items, and use simple manually designed interaction functions (e.g., logistic function in IRT). To improve the precision and interpretability, previous works mainly focus on the representation learning of trait features (e.g., extending trait features into multidimension [Reckase, 2009]) and interaction function design (e.g., using neural networks to automatically learn the complex interaction function [Wang et al., 2020]). To preserve the monotonicity, previous works tend to apply a monotone function as the interaction function rather than considering it in the optimization criteria.

∗Corresponding Author.

In the literature, the monotonicity theory assumes that student's proficiency is monotonic with the probability of giving the right response to a test item [Rosenbaum, 1984; Wang et al., 2020]. As shown in Figure 1, we can easily find that Ada with a right response to item e1 is considered as having a higher proficiency level on the related concept A (i.e., Function) than Bob with a wrong response. This observation can be further interpreted from a monotonous point of view that students with correct responses should be more proficient than students with wrong responses. Therefore, based on the partial order between item responses, we can create a new optimization criteria to exploit the response pairs to enhance the monotonicity.

However, how to exploit the partial order between responses to improve the monotonicity within the optimization criteria is a challenging problem. First, it is hard for us to compare item responses across different items related to nonoverlapped concepts. For example, as shown in Figure 1, it is hard to tell whether Ada is more skilled than Bob on concept B (i.e. Derivate) and E (i.e., Equation) when we only observe that Ada gives a wrong response to item e5 and Bob gives a right response to item e2. Second, there exist many unobserved responses such as Bob not giving a response to item e5 in Figure 1. Last but not least, how to find an objective function so that the monotnicity can be directly optimized is another important challenge.

To this end, we propose the general Item Response Ranking framework (IRR) for cognitive diagnosis, which can be applied to most of contemporary CDMs. Specifically, we first design an item specific pair sampling method to resolve the potential non-overlapped problem, i.e., sampling responses from different students to the same item to keep related knowledge concepts the same. Then, to handle the unobserved responses along with the observed responses, we conduct a two-branch sampling method, i.e., positive sampling and negative sampling. After that, based on the sampled pairs, we introduce the pairwise learning to model the partial order among response pairs, where we use a pairwise objective function to better optimize the monotonicity. Extensive experiments on two real-world datasets show that CDMs with IRR not only significantly outperforms the baselines, but also effectively provides interpretable insights for understanding the cognitive diagnostic results of students.

2 RelatedWork

2.1 Cognitive Diagnosis

Cognitive diagnosis is a fundamental but important task in many real-world scenarios such as games[Chen and Joachims, 2016], medical diagnosis [Xu et al., 2017], and especially, in education [Liu et al., 2018; Liu et al., 2019a; Huang et al., 2020]. IRT [Lord, 1952] and DINA [De La Torre, 2009] are the two most fundamental but classic cognitive diagnosis models, which model the response result of a student answering an item as the interaction between the trait features of the student and the item. By extending the trait features into multidimensional, Reckase et al. [2009] proposed Multidimensional Item Response Theory (MIRT). Noticing the manually designed interaction functions of previous works may restrict the model scope of applications, NeuralCD [Wang et al., 2020] exploited neural networks to automatically learn the interaction function. However, few of these works pay enough attention to the monotonicity, especially during optimization, which somehow limits their performance.

2.2 Pairwise Learning

Pairwise learning is an approach to explore the relations in a pair, which has been widely used in many areas (e.g., recommendation [Rendle et al., 2012], natural language process [Huang et al., 2017] and computer vision [Wang et al.,

2017]). For instance, Huang et al. [2013] used pairwise learning strategy to facilitate the training process for solving information retrieval. Rendle et al. [2012] constructed training pairs to solve the ranking problem in recommender system with implicit feedback. However, few works have been taken to integrate these methods into cognitive diagnosis.

3 ProblemDefinition

Let S = {s1, s2, ..., sN} be the set of all N students, E = {e1, e2, ..., eM} be the set of all M test items and K = {k1, k2, ..., kL} be the set of L knowledge concepts. Suppose the response logs R are denoted as set of triplet (s, e, r), where s ∈ S, e ∈ E and r is the score (transferred to binary, i.e., 0 indicates wrong answer while 1, otherwise). For convenience, we also denote R as rse. Furthermore, we have Q-matrix [Tatsuoka, 1995] labeled by experts, Q = {Qij}M×L, where Qij = 1 if the item ei relates to the knowledge concept kj and Qij = 0 otherwise.

Definition 1. Cognitive Diagnosis: Given response logs R and Q-matrix Q, our goal is to mine students' proficiency on knowledge concepts.

4 Preliminary

Before we step into our method, we would like to first briefly introduce Cognitive Diagnosis Models (CDMs). CDMs are developed to depict student's proficiency level on specific knowledge concepts based on her responses to several test items. To do this, a pointwise objective function is used to train CDMs on the Student Performance Prediction task. More concretely, CDMs are expected to minimize the difference of the predicted probability P(yie) of a student i giving the right response to the item e between the true response rie:

In the past decades, lots of CDMs have been proposed such as DINA and IRT. Generally, CDMs contain two parts: (1) the representations of trait features and (2) the interaction function. For example, IRT uses single-dimension variables to represent the trait features and logistic function as the interaction function as follows:

where a and b represent the discrimination and difficulty of item e, and θ indicates the proficiency level of the student i. Using multidimensional vectors to represent latent traits of both test items and students, IRT is extended to MIRT:

However, CDMs trained by Eq. (1) cannot involve the optimization of the monotonicity while we argue that the monotonicity should be included during optimization.

5 ItemResponse Ranking

5.1 Overview

As we discussed before, the referred monotonicity declares that the student's proficiency is monotonic with the probability of giving the right response to a test item, which could be further expressed in a pairwise perspective: a more skilled student should have a higher probability to give the right response to a test item than an unskilled one. Formally, we have the following pairwise monotonicity:

Lemma 1. Pairwise Monotonicity Given a specific test item, the students with right responses are more skilled than those with wrong responses.

However, as we mentioned above, traditional methods use the pointwise optimization objective function (i.e., Eq. (1)) to train CDMs, which cannot optimize the monotonicity between responses. Therefore, instead of minimizing the difference of the predicted probability P(yie) between the true response rie, we would like to directly optimize the pairwise monotonicity in the objective function, i.e.,

where i and j represent different students.

In the following paragraphs, we will introduce: (1) how to construct the training pairs and (2) how to design the objective function so that the pair monotonicity can be promised.

5.2 Pair Construction

An important issue in IRR is to construct training pairs. For each response triplet R = (u, e, r) ∈ R, we want to sample some triplets to form training pairs T(R). In the following paragraphs, we will first conduct an item specific sampling, where we divide the students based on the item they give the response to. Then we will show how we handle the students with unobserved response, where we propose a two-branch sampling method. In the last part, we will present how to perform the training pair sampling.

Item Specific Sampling

For each item e, we first divide the students into two groups, one containing students who have completed the project and the other containing those who have not completed yet. The former one is called observed students SO (e) and the latter one is called unobserved students SU (e). Furthermore, we divide SO (e) into two subgroups based on their performance on e: (1) positive students S+(e) are those who correctly answer e and (2) negative students S-(e) are those who give the wrong answer:

Two-branch Sampling

Different from other scenarios like recommendation [Rendle et al., 2012] where the unobserved data is usually treated as a negative sample, in cognitive diagnosis, the unobserved response log should not be simply treated as a negative one. Intuitively, we assume: the probability of any unobserved student u correctly answering e is 0 ≤ P(rs-e) ≤ P(rue) ≤ P(rs+e) ≤ 1, which can be divided into two parts:

where s- ∈ S-(e) and s+ ∈ S+(e). As shown in Figure 2, based on Eq. (6) and Eq. (7), we have the two-branch sampling, i.e., positive sampling and negative sampling:

Positive sampling: Based on Eq. (6), for s- ∈ S-(e), those unobserved students are considered as positive samples, which is shown in the right-top part of Figure 2.

Negative sampling: Based on Eq. (7), for s+ ∈ S+(e), those unobserved students are considered as negative samples, which is illustrated in the right-bottom part of Figure 2.

Training Pair Sampling

The total number of training pairs is |S+(e)×(S-S+(e))|+ |S-(e)×(S-S-(e))|, which is quite large. To accelerate the training speed, we apply the sampling method. Specifically, for each response triplet R = (s, e, r), we randomly select at maximum of N O observed samples and at maximum of N U unobserved samples during each training step. If the student u in the response triplet R = (s, e, r) is a positive student (i.e., s ∈ S+(e)), the observed samples are selected from S-(e) and unobserved samples are selected from SU (e). We similarly select samples for the negative students and we select at maximum of N O observed samples and at maximum of N U unobserved samples to form training samples T(R):

where Λx (S) means randomly selecting at maximum of x elements from the set S to form a subset and ⊕ is the set addition. The discussion of how the sample number (i.e., N O and N U ) affects the model performance is shown in Section 6.4.

5.3 LearningModel with IRR

After we construct the training pairs T(R) for each response triplet, we are going to talk about the objective function.

Firstly, the log-likelihood of IRR is:

× (1 - P (rje ≤ rie)).

Secondly, the objective function of IRR is:

where λ(Θ) is the regularization term and λ is a hyperparameter. With the pair sampling mentioned in 5.2, we rewrite Eq. (12) as the following loss function:

We can apply IRR to any fully differentiable CDMs (e.g., MIRT) and train them with Stochastic Gradient Descent.

6 Experiments

In this section, we first introduce the datasets and our experimental setups. Then, we conduct extensive experiments to compare the performances of CDMs optimized by the pointwise approach and our IRR (hereinafter referred to as pointwise-CDMs and IRR-CDMs respectively) to answer the following questions:

• RQ1: Can IRR-CDMs perform better in predicting student performance and preserve the monotonicity compared to the pointwise-CDMs?
• RQ2: Are the diagnostic results of IRR-CDMs monotonic on the knowledge level?
• RQ3: How does the number of samples (i.e., N O and N U ) influence the performance of IRR-CDMs?
• RQ4: What are the differences of the diagnostic results between IRR-CDMs and pointwise-CDMs?

Our code is available at https://github.com/bigdataustc/EduCDM.

6.1 Dataset Description

We use two real-world datasets in the experiments, i.e., ASSISTments and MATH. ASSISTments (ASSISTments 20092010 "skill builder") is an open dataset collected by the ASSISTments online tutoring systems [Feng et al., 2009]. Collected from a widely-used online learning system, MATH contains mathematical test items and logs of high school examinations. Table 1 shows basic statistics of the datasets.

We filter out students with less than 15 and 30 response logs for ASSISTments and MATH respectively to guarantee that each student has enough data for diagnosis. For each dataset, we divide the students on each test item into training: test = 8:2. We use 90% of the training data to train model and apply grid search to adjust the hyper-parameters on the remaining 10% of the data (i.e., the validation dataset).

6.2 Experimental Setup

To evaluate the performance of our IRR, we apply our framework to four well-known CDMs, i.e., DINA, IRT, MIRT and NeuralCD. We make a statistics on the overall dataset to get the correct portion ce of each item and the CDMs with IRR predict the top ce percentage of students on each item giving the right responses. In multidimensional models (i.e., MIRT and NeuralCD), we set the dimension of latent trait features of both student and item unitedly as the number of knowledge concepts, i.e., 123 in ASSISTments and 1488 in MATH. All hyper-parameters are tuned in the validation datasets. λ is selected from [0.1, 0.01, 0.001, 0.0001]. N O and N U are selected from [1, 5, 10, 30]. Based on the performance on the validation datasets, we set λ = 0.0001 and N O = N U = 10.

We initialize parameters in all networks with Xavier initialization [Glorot and Bengio, 2010] and we use the Adam algorithm [Kingma and Ba, 2014] for optimization. All models are implemented by MXNet using Python and all experiments are run on a Linux server with two Intel(R) Xeon(R) E5-2699 v4 CPUs and a Tesla P100 PCIe GPU.

6.3 EvaluationMetrics

Classification and Ranking Metrics. Because we cannot obtain the true knowledge proficiency of students, it is hard to directly evaluate the performance of a cognitive diagnosis model. Following previous works [Wang et al., 2020], as the diagnostic result is usually acquired through students performance prediction task, performance on the prediction task can indirectly evaluate the model based on some classification metrics such as AUC, Precision, Recall and F1. Besides, we need some metrics to investigate whether the monotonicity is maintained in models. Based on the monotonicity, we hope the models can correctly assign a higher predicted score to the the more skilled student. This is quite similar to a ranking problem. Therefore, we apply some commonly used ranking metrics [Yu et al., 2018] to evaluate the monotonicity. The ranking metrics include MAP and NDCG@k. However, it is worth noticing that, we should not only focus on the top students (i.e., excellent students), but also need to pay attention to the underachievers (i.e., ranked at the bottom). Therefore, rather than evaluating the ranking accuracy result based on the descending order, we also evaluate it based on the ascending order. For example, given a descending ranking list [r1, ..., rN] where the fronted students are predicted to have higher probability to correctly answer an item and r is true score, traditional NDCG@k is calculated by NDCG@k = NDCG(r1, ..., rk). The inverse version NDCG is defined as INDCG@k = NDCG(-rN, ...,-rN-k+1). For convenience, we denote the top metrics as MAP(E), NDCG@k(E) and Precision@k(E), and represent the inverse version of previous metrics as NDCG@k(U), NDCG@k(U) and NDCG@k(U), where E and U respectively represent Excellent students and Underachievers.

Table 2: Experimental results on student performance prediction.

Degree of Agreement. Following Wang et al. [2020], we adopt Degree of Agreement (DOA) to further investigate the monotonicity based on concepts. Specifically, if student i has a better mastery on knowledge concept k than student j, then i is more likely to answer item l related to k correctly than j. For concept k, DOA(k) is formulated as:

dent i on concept k. δ(x, y) = 1 if x > y and δ(x, y) = 0 otherwise. Ilk = 1 if item l contains concept k and Ilk = 0 otherwise. J(l, i, j) = 1 if both student i and j did item l and J(l, i, j) = 0 otherwise. We average DOA(k) on all concepts (i.e., DOA) to evaluate the quality of diagnostic result.

6.4 Experimental Results

Student Performance Ranking (RQ1)

The experimental results are shown in Table 2. Each CDM result has two sub-columns, i.e., the left one shows the performance of the pointwise-CDM and the right one presents the IRR-CDM. Wilcoxon rank-sum statistical tests [Yu et al., 2018] have been used to check whether the difference between original optimization strategy and our method are statistically significant (with a 0.05 significance level). From the table, we can see that for every CDM, the IRR-CDM significantly outperform the pointwise-CDM on classification metrics on all datasets. This indicates that our proposed framework can be applied to most of contemporary CDMs and can promote the diagnosis precision. Besides, we notice that, IRR-CDM also achieve higher scores in ranking metrics which means IRR can help CDMs to better maintain the monotonicity during training. Summarily, we have the conclusion that by exploiting the partial order between responses, IRR can not only help CDMs promote the diagnosis precision but also better maintain the monotonicity.

Knowledge Proficiency Monotonicity (RQ2)

Among CDMs, we only use DINA and NeuralCD as base models to verify whether the diagnostic results are monotonic on knowledge level, since for IRT, MIRT, there are no clear correspondence between their latent features and knowledge concepts. Figure 3 presents the experimental results. From the figure, we can observe that DOAs of IRR-CDMs are higher than pointwise-CDMs, which proves that the knowledge proficiency diagnosed by IRR is more monotonic.

Performance with Different Sample Number (RQ3)

In IRR, the number of samples (i.e., N O and N U ) plays a crucial role. We use IRT and MIRT as base models to investigate the influence of different number of samples on the effectiveness of IRR. For convenience, we set N U = N O and conduct the experiment by assigning different N O from set {1, 2, 3, 4, 5, 10, 30}. As shown in Figure 4, as the sample number increases, the performance of IRR-CDMs increases at the beginning, but it converges afterwards both in two datasets. Meanwhile, we can find that compared with MATH, the performance of IRR-CDMs on ASSISTments converges earlier. The reason might be that the ASSISTments contains less data which means the models need fewer sampled data to acquire enough information. These results indicate that including more sampled pairs can help the IRR-CDMs promote the performance, but the promotion has a upper bound due to the converge of information increment.

Diagnostic Results Analysis (RQ4)

Here we present an example of a student's diagnostic results on dataset ASSISTments in Figure 5 to show the differences between pointwise-CDMs and IRR-CDMs, and we use NeuralCD as the base model. Figure 5 (a) shows the response logs of three students on three items and the related knowledge concepts of each item. Figure 5 (b) presents the cognitive diagnostic results from the pointwise-CDM (the left part) and IRR-CDM (the right part). We first look at the diagnosis report from the IRR-CDM. We can find that with correctly answering item 1 and item 2, which both contain the concept B (i.e., Circle Graph), student 2 has a higher proficiency on the concept B than student 1 and student 3. Similarly, we can observe that student 2 and student 3 are diagnosed to be more skilled on concept E (i.e., Finding Percents) than student 1, where student 2 and 3 correctly answer the related item 2 while student 1 not. Compared with the result in the left part, we can find that IRR-CDMs get a more precise and more discriminated ranking result.

Meanwhile, comparing the distribution of proficiency values in two diagnosis reports in Figure 5 (b), we notice that IRR-CDMs can give more discriminated diagnosis values. For better illustration, we visualize the all diagnosis proficiency values on concept B (Circle Graph) and concept D (Equivalent Fractions) in Figure 5 (c). We can find the distributions of the diagnosis proficiency values got by IRR-CDMs are more smooth, which means the proficiency values are

(b) Proficiency on knowledge concepts digram.

(c) Proficiency distribution digram.

Figure 5: An example of diagnostic results.

more discriminated. Based on these observation, we have the conclusion that our method can help CDMs get a more discriminated diagnosis proficiency values, which accounts for why IRR-CDMs can well perform in cognitive diagnosis.

7 Conclusion and FutureWork

In this paper, we present the monotonicity in a pair formulation and proposed the Item Response Ranking framework to incorporate the monotonicity into the optimization objective. With the proposed item specific two-branch sampling method, we manage to introduce the pairwise learning into cognitive diagnosis. As a general framework, IRR can be applied to most of CDMs. Extensive experiments demonstrate the effectiveness and interpretability of IRR framework.

In the future, we are going to optimize the response prediction with IRR by introducing difficulty prediction and fuzzy algorithm to better approximate the real correct portion. Meanwhile, we tend to improve the sampling strategy to find more distinguished samples during training so that the IRR can be more effective and efficient.

Acknowledgements

This research was partially supported by grants from the National Natural Science Foundation of China (Grants No. 61922073 and U20A20229) and the Fundamental Research Funds for the Central Universities (Grants No. WK2150110021).