A Fast-Adaptive Cognitive Diagnosis Framework for Computerized Adaptive Testing Systems

1 Introduction

With the rise of online testing platforms, the demand for personalized assessments has surged. Standardized exams like the GMAT and GRE have already used adaptive question selection strategies to better assess students' abilities. Computerized Adaptive Testing (CAT) [Mills and Steffen, 2000; Rudner, 2009; Liu et al., 2024] optimizes test efficiency by tailoring questions to each student, reducing the number of questions needed while still ensuring accurate ability assessments. Unlike traditional tests, the Cognitive Diagnosis Model (CDM) in CAT estimates a student's current ability based on their responses to the questions selected by the selection strategy and provides feedback for further question selection. Consequently, CDM [Liu, 2021; Wang et al., 2024; Li et al., 2025] serves as a fundamental and decisive component in CAT [Anderson et al., 2014].

∗Corresponding Author.

In recent decades, significant efforts have focused on developing CDMs, which model the interaction between students and questions. Traditional CDMs [Embretson and Reise, 2013; De La Torre, 2009; Sympson, 1978] rely on expert-crafted interaction functions, while neural networkbased CDMs [Wang et al., 2020a; Wang et al., 2023a; Ma et al., 2022; Li et al., 2022; Liu et al., 2023] use multilayer perceptrons to enhance generalization and interpretability. Recently, the use of graph neural networks (GNNs) to learn response relation graphs has gained attention, offering new insights for cognitive diagnosis [Qian et al., 2024; Gao et al., 2021; Shen et al., 2024a; Wang et al., 2023b].

Although the aforementioned CDMs have made remarkable progress, few studies focus on their adaptability to downstream tasks like CAT, which heavily relies on the assumption of relatively accurate CDM's outcomes. The goal of CAT is to assess a student's abilities with the fewest possible questions, posing a key challenge that CDM should be able to dynamically model students' abilities for rapid adaptation during the early stage of CAT, however the early stage suffers from limited data, which is not conducive to this rapid and accurate adaptation. As shown in Figure 1, we compare the performance of Item Response Theory (IRT) [Embretson and Reise, 2013] and Neural Cognitive Diagnosis Model (NCD) [Wang et al., 2020a] in real CAT tasks using BECAT [Zhuang et al., 2023] for selection. IRT's simplicity limits its adaptability, resulting in slow performance improvements. Although NCD captures complex response patterns, its reliance on sufficient data leads to poor early-stage performance, conflicting with CAT's goal of rapid and accurate assessment.

To this end, this paper proposes a fast-adaptive cognitive diagnosis framework for computerized adaptive testing systems (FACD) to address the challenge mentioned above and effectively enhance CDM's adaptability especially during the early stage of CAT. Specifically, for the overall dynamic structure modeling, a dynamic collaborative diagnosis module are designed. Unlike traditional CDMs which rely on static response logs, the proposed FACD framework dynamically updates a response graph at each CAT stage to transfer relevant student and concept information, enabling rapid construction of the student's cognitive profile. For modeling personal dynamic response sequences, we design a dynamic personalized diagnosis module that employs a Gated Recurrent Unit (GRU) [Chung et al., 2014] to capture the relationships in each student's response sequence. A self-attention mechanism further highlights critical response features. The collaborative information and the personalized information are then fused to model the ultimate representation and fed into the subsequent interaction function. Extensive experiments show that FACD improves early-stage performance in CAT by about 5%-10% compared with traditional CDMs.

The subsequent sections respectively recap the related work, present the preliminaries, introduce the proposed FACD, show the empirical analysis and conclude the paper. We include a section on the statement on the ethical use of data and informed consent of research subjects in Section 6.

2 RelatedWork

2.1 Computerized Adaptive Testing

CAT consists of two key components: a CDM and a selection strategy. Traditionally, IRT [Embretson and Reise, 2013] and NCD [Wang et al., 2020a] are popular choices for CDM in CAT. Nowadays, most of the researches focus on the selection strategy in CAT. The most commonly used approaches like Maximum Fisher Information (MFI) [Lord, 2012] and Kullback-Leibler Information (KLI) [Chang and Ying, 1996] are typically tailored to specific CDMs, such as IRT. To address this limitation, Model-agnostic algorithms like MAAT [Bi et al., 2020] and BECAT [Zhuang et al., 2023] are introduced, but they rely on accurate CDM diagnosis, which is challenging in the early stages of CAT. Inaccurate CDM results can mislead selection strategies, delaying the whole CAT process. Data-driven strategies like BOBCAT [Ghosh and Lan, 2021] and NCAT [Zhuang et al., 2022a] pretrain strategy models on large datasets, but still struggle with the instability of early-stage CDM diagnosis. Thus, a CDM tailored to CAT is essential for accurate and timely early-stage diagnosis.

2.2 Cognitive Diagnosis

Traditional cognitive diagnosis infers students' mastery levels of concepts using models like IRT [Embretson and Reise,

2013], MIRT [Sympson, 1978], and DINA [De La Torre, 2009]. With deep learning advances, models using multilayer perceptrons (MLPs) like NCD [Wang et al., 2020a], KSCD [Ma et al., 2022] and KaNCD [Wang et al., 2023a] have successfully captured complex interactions. Other approaches to extracting rich information from response logs include symbolic regression [Shen et al., 2024b], graph attention networks [Gao et al., 2021; Qian et al., 2024; Shen et al., 2024a; Wang et al., 2023b] and Bayesian networks [Li et al., 2022]. Recently, several CDMs have been developed to address cold-start issues in static CD, such as TechCD [Gao et al., 2023] and ZeroCD [Gao et al., 2024], which focus on cross-domain CD, and BetaCD [Bi et al., 2023] and AGCDM [Pei et al., 2022], which tackle limited interaction data for specific students in CD. While these issues share some similarities with the cold-start problem in the early stage of CAT, they are not directly applicable due to dynamic context in CAT. Furthermore, few CDM studies focus on adapting to downstream tasks, and to our knowledge, no CDM has been specifically designed for CAT's unique requirements, making such adaptation crucial.

3 Preliminaries

Cognitive Diagnosis Basis. Consider an educational system with N students, M questions and K knowledge concepts, which can be denoted as three sets: S = {s1, ..., sN}, E = {e1, ..., eM} and C = {c1, ..., cK}. The relationship between the questions and the knowledge concepts can be recorded in a binary Q-matrix Q = {Qij}M×K where Qij ∈ {0, 1} indicates whether the knowledge concept is involved in the question or not. Each student in S will selectively do some questions from the set E, generating response logs that record their performance. These response logs are illustrated as triplets R = {(s, e, r)|s ∈ S, e ∈ E, rse ∈ {0, 1}} with rse = 1 meaning question e is correctly answered by student s while rse = 0 standing for the opposite. Based on the messages above, cognitive diagnosis aims to infer Mas ∈ RN×K , a matrix that indicates each student's mastery level across all knowledge concepts.

Computerized Adaptive Testing. In CAT, the selection strategy and CDM alternate in a cyclical process. For the CDM, let us define the CDM as M. And a student i's question set Ei can be divided as candidate and evaluated questions set as Ci and Ji. Assuming at time step t ∈ [1, T], student i has already answered a sequence of questions Rt-1,i = {(e1, r1), ..., (e|t-1|, r|t-1|)} selected from candidate set Ci. Then, the CDMM can estimates student i's mastery level at time t based on the response sequence Rt-1,i, denoted as ˆ Mast-1,i =M(Rt-1,i) ∈ [0, 1]. For the selection strategy, let us define the selection policy as π which selects a question et,i ∼ π( ˆ Mast-1,i) for student i. Upon receiving student i's response rt,i, the CDM is updated, resulting in a new mastery level estimate ˆ Mast,i. And after updating, the performance of the CDM is tested on the evaluated questions set Ji. The above process is repeated T steps. The goal of CAT is to let the student estimated mastery level ˆ MasT,i close to the real value Mas∗i with as few test steps T as possible.

Problem Statement. Our paper primarily focuses on building a fast-adaptive CDM module to help the CDM in CAT systems estimate a student's mastery level rapidly and stably, bringing it closer to the real value as soon as possible. Specifically, our goal is to ensure that at the relatively early time step T, the estimated mastery level ˆ MasT,i = M(RT-1,i) is as close as possible to the real mastery level Mas∗i . However, since we cannot directly observe Mas∗i , our goal shifts to estimate the distribution of the student's actual responses R∗ i . The closer the estimated response distribution RT,i is to the real response distribution R∗ i , the better the CDM's performance. This difference is typically measured using Area Under the Curve (AUC), where a higher AUC( ˆRT,i, R∗ i ) indicates more accurate predictions from the CDM.

4 Methodology

In this section, we provide the technical details of the proposed FACD, depicted in Figure 2. We introduce two crucial components: a dynamic collaborative diagnosis module in Figure 2(a) and a dynamic personalized diagnosis module in Figure 2(b). Following above, we will describe how to integrate these features into the interaction functions of various CDMs in the ultimate inference module. Detailed components of FACD are shown in Figure 2(c).

4.1 Dynamic Collaborative Diagnosis Module

Graph Construction. Firstly, we define the dynamic response graph at time step t for dynamic collaborative diagnosis module, denoted as Gt = (V, Et). The node set V = S ∪ E ∪ C involves students, questions, and concepts nodes respectively. V remains unchanged during the CAT process, with all nodes being defined initially. The edge set Et = E se t ∪ E ec consists of two types of edges where E se t involves the interactions between students set S and questions set E (i.e., student si complete the question ej) as well as E ec involves the relationships between questions set E and concepts set C (i.e., question ej is related to the concept ck). Based on this, we can decompose the graph into two subgraphs. The first subgraph is the student-question graph, which is dynamic in nature. At each time step t, the edge set is updated as E se t = E se t-1 ∪ Rt, where Rt represents the newly added interaction records of all students at time step t. The corresponding adjacency matrix Ase t ∈ R|S|×|E| , can be expressed as follows:

The second subgraph is the question-concept graph, which is a static graph. The edge set E ec can be directly derived from the relationships between questions and concepts in the Q-matrix. The corresponding matrix Aec ∈ R|E|×|C| , can be expressed as Aec = Q. Then we can construct the final dynamic response graph Gt which integrates the above two subgraphs. And its corresponding matrix At ∈

R(|S|+|E|+|C|)×(|S|+|E|+|C|) can be expressed as follow:

where O means zero matrix. The graph Gt dynamically creates connections between students and questions during question selection in CAT and construct the relationship between question and concept, which is fundamental to the subsequent extraction of collaborative information.

Graph Convolution. In CDM, the fundamental data elements comprise response logs and the Q-matrix. It is necessary to systematically deconstruct these intricate logs into their essential components: students, questions, and concepts. To represent these entities, we employ trainable embeddings, denoted as Z(0) s ∈ R|S|×d ,Z(0) e ∈ R|E|×d ,Z(0) c ∈ R|C|×d where d indicates the dimension of the embeddings, as initial ID representation. And to facilitate subsequent convolution processes, we stack the aforementioned embeddings to form Z(0) ∈ R(|S|+|E|+|C|)×d , so the initial dynamic collaborative representation at time step t = 0 can be defined as Z(1) 0 = Z(0) . Given that the representation of students, questions, and knowledge concepts in CDM are relatively straightforward and consist solely of IDs. Inspired from [He et al., 2020], we eliminate linear transformations and nonlinear activation functions, choosing instead to focus on the fundamental elements of graph convolutional networks (GCNs). Therefore, the graph embedding propagation layer at time step t is formulated in the following matrix representation:

where the degree matrix D is a diagonal matrix with size (|S| + |E| + |C|) × (|S| + |E| + |C|), and each entry Dii indicates the number of non-zero entries in the i-th row vector of the adjacency matrix At. By applying Eq. (3), we can derive the convolution results Z(1,l) t ∈ R(|S|+|E|+|C|)×d from the l-th layer of graph Gt at time step t. The dynamic collaborative representation ˜Z(1) t ∈ R(|S|+|E|+|C|)×d at time step t is calculated through a mean pooling operation on the outcomes from each layer of graph Gt which can be expressed as ˜Z(1) t = 1 1+L ∑Ll=0 Z(1,l) t .

The graph convolution process aggregates outcomes from neighbors which is crucial for extracting collaborative information, as it enables us to identify patterns and relationships that reflect the collective strengths and weaknesses on student's mastery level within their similar neighbors. This helps us rapidly construct a profile for the student based on similar student groups in the early stage of CAT.

Graph Fusion. Given the inherent instability associated with training dynamic graphs, we fuse both the dynamic collaborative representation ˜Z(1) t ∈ R(|S|+|E|+|C|)×d of current time step t and the averaged dynamic collaborative representation Z(1) t ∈ R(|S|+|E|+|C|)×d from previous time steps 0 to t-1 in constructing the ultimate dynamic collaborative representation Z(1) t ∈ R(|S|+|E|+|C|)×d . This approach mitigates the potential training instability arising from variations in the dynamic graph, which can be expressed as follows:

where ηt is the smoothing weight for the dynamic collaborative features which can either be set using a predefined threshold to determine the fusion ratio, or be adjusted adaptively through a learning mechanism. We currently set ηt = 0.5.

4.2 Dynamic Personalized Diagnosis Module

For every student si, we construct a dynamic personalized representation at time step t based on their response records of the selected questions. To capture the sequential relationships between questions chosen by the selection algorithm in CAT for each student, we utilize the GRU network [Chung et al., 2014]. The personalized question representation sequences Lt,si ∈ Rt×d for student i at time step t are defined as follows:

where ht,si ∈ R1×d denotes the hidden states of questions selected at the time step t in GRU. Then we utilize the multihead self-attention over the personalized question representation sequences Lt,si , to extract the most critical response information of each student and get the dynamic personalized representation Z(2) t,si ∈ R1×d of student si at time step t:

where WQ h ,WK h ,WV h , ∈ Rd×dh are the trainable parameter matrices for the query matrix, key matrix and value matrix respectively, WF ∈ Rd×d , bF ∈ R are the trainable parameters for the final linear transformation, dh = Hd denotes the dimension of each self-attention head, H stands for the number of self-attention heads, and softmax(·) and [·] is the row-wise softmax function and concatenation operation respectively.

Compared to traditional CDMs that treat the student, question and response records as discrete data points for training, our dynamic personalized diagnosis module offers a sequential modeling of questions for each student. This approach emphasizes the interrelationships among questions within the CAT selection strategy while extracting pertinent response information for each student using self-attention. Consequently, it lays a foundation for personalized modeling and enhances the robustness of CDMs in fast adaptation of CAT.

4.3 Inference Module

Representation Fusion. After obtaining the representations from the dynamic collaborative diagnosis module and dynamic personalized diagnosis module, the fusion module uniformly models the ultimate fused representation for each student si, each question ej and each concept ck at time step t. Specifically, we can express this fusion process as follow:

where Ht,si ∈ R1×d , Ht,ej ∈ R1×d and Ht,ck ∈ R1×d denote the ultimate fused representation for each student si, each question ej and each ck at time step t respectively. Z(0) si , Z(1) t,si and Z(2) t,si represent the initial ID representation, dynamic collaborative representation and dynamic personalized representation as mentioned above. And αt is the normalized weight of three kinds of representation. Taking αt,si as example, we first calculate these weights as:

where W0 s ∈ R3d×1 , W1 s ∈ R3d×1 , W2 s ∈ R3d×1 , b0s ∈ R, b1s ∈ R, b2s ∈ R are trainable parameters with concatenation [·] for three kinds of representation. After that, we normalize the weights as follow:

As for question and concept fused representations, the process is similar to the student.

Interaction Function. Given input representation Ht,si , Ht,ej , Ht,ck , our model is capable of adapting to various interaction functions present in existing CDMs to predict the performance of students' practicing questions, which can be formulated as ˆ yij = M(ϕs(Ht,si ), ϕe(Ht,ej ), ϕc(Ht,c)), where M(·) can be the interaction functions in any kinds of CDMs, and ϕ(·) is the transformation layer which facilitate the integration of representation with the majority of interaction functions in existing CDMs and help transform the representation's dimensions to suit the feature space in specific type of CDM. For simplicity, we use the linear transformation in our framework.

Model Training. In the CD task, we primarily employ the binary cross entropy loss function, which measures the difference between the model's predicted outcomes and the actual response scores within a mini-batch. And in CAT process, the loss is calculated on each student's response set Rt of the newly selected question set at time step t. The loss function can be formulated as:

5 Experiments

5.1 Experimental Settings

Dataset Description. The experiments are conducted on three real-world datasets, i.e., FrcSub [DeCarlo, 2011; Tatsuoka, 1984], EDMCup2023 [Ethan Prihar, 2023] and NeurIPS2020 [Wang et al., 2020b]. The details about datasets source and data preprocessing are depicted in the Appendix D.1. The Appendix A-D can be found in https: //github.com/BW297/FACD.

Evaluation Metrics. Evaluating the performance of CDM in CAT systems is inherently difficult due to the challenges in accurately assessing real students' mastery level Mas∗ . To tackle this issue, a common approach is to predict students' response scores. We measure the model's ability to predict whether a student successfully solves an question compared to the actual outcomes in the test set, utilizing standard classification metrics such as accuracy (ACC) and area under the curve (AUC).

Implementation Details. For parameter initialization, we employ Xavier [Glorot and Bengio, 2010], and for optimization purposes, Adam [Kingma and Ba, 2015] is adopted. The batch size is set within the range {32, 64, 128, 256}. The learning rate is chosen from {1e-3 , 3e-3 , 5e-3 , 7e-3 , 1e-2 }. The dimensions of the MLPs for all methods are consistent, being 512 and 256. To ensure robustness and reproducibility, each experiment is repeated ten times. The source code for our implementation can be found in https://github.com/ BW297/FACD. More details of the implementation can be found in Appendix D.3.

5.2 Performance Comparison

To showcase the effectiveness of FACD, we integrate it with both IRT [Embretson and Reise, 2013] and NCD [Wang et al., 2020a], which are widely utilized in CAT researches [Bi et al., 2020; Zhuang et al., 2022a; Zhuang et al., 2023], as discussed in the previous sections. The detailed introduction of baselines used in experiments below is provided in Appendix D.2.

Performance Comparison with Different CAT Selection Strategies. To demonstrate the strong compatibility of our method with various CAT selection strategies, we have chosen a set of both classical and cutting-edge strategies as follows: Random, MAAT [Bi et al., 2020], BOBCAT [Ghosh and Lan, 2021], NCAT [Zhuang et al., 2022b] and BECAT [Zhuang et al., 2023].

As shown in Table 2, it is evident that various CDMs demonstrate strong adaptability when integrated with FACD across all CAT selection strategies. Notably, during the initial stages of question selection, such as after answering approximately five questions, our model exhibits fast adaptation, creating a significant performance gap compared to the original CDMs. In later stages of question selection, the performance improvement of our model becomes more incremental, as it approaches the student's true mastery level. This highlights the model's ability to converge within a limited number of steps, providing precise feedback to CAT selection strategies and supporting a more efficient reduction in the number of questions selected.

Performance Comparison with Different Extraction Modules in CDM. To compare different representation extraction methods, we use BECAT as selection strategy and select several baselines for comparison with FACD as follows: RCD [Gao et al., 2021] and ORCDF [Qian et al., 2024].

As shown in Table 3, our FACD consistently outperforms RCD and ORCDF in CAT system and demonstrates relatively stable results across different CDMs. This is primarily because RCD and ORCDF lack stability research for dynamic graph modules. Furthermore, they fail to effectively model the personalized sequence of questions for each student and thus do not assess the importance of student response information, making them sensitive to noise, such as low-quality response data, which reduces their robustness. It can be observed that FACD not only ensures strong performance and robustness but also effectively mitigates BECAT's cold start issue as mentioned in original paper [Zhuang et al., 2023]. We also provide further comparison with traditional classical GNN frameworks in Appendix D.5.

5.3 Ablation Study

To showcase the contributions of each component in FACD, we conduct an ablation study on FACD in dataset EDMCup2023, which is divided into the following three versions. "FA-C" and "FA-P" means removing dynamic collaborative diagnosis module and dynamic personalized diagnosis module, respectively. "OL" means the base CDMs. As shown in Table 4, the proposed method outperforms the other two ablated versions, indicating that each component contributes significantly to different stages of CAT, resulting in the overall effectiveness of the model. Early performance relies on the dynamic collaborative diagnosis, while later stability comes from the dynamic personalized diagnosis.

5.4 In-Depth Analysis of FACD's Advantages

Inference Time Comparison. We compare the inference speeds of different extraction modules and the original CDMs on the EDMCup2023 dataset, specifically the average CPU time per round spent on training and inference after the CAT

(c) Pretrain Ratio 0.2

(d) Pretrain Ratio 0.3

Figure 4: Pretraining ratio analysis.

system selects a question for each student. As shown in Figure 3, RCD, using complex graph aggregation functions, has slower inference speeds and higher resource consumption. Both FACD and ORCDF, using lighter aggregation methods similar to LightGCN [He et al., 2020], are faster, but ORCDF incurs additional costs due to its more complex response graph. Our method achieves high-quality cognitive diagnosis with limited computational resources, making it well-suited for modern online education platforms. Detailed time and space complexity analysis is provided in Appendix B.2.

Generalization Analysis on Pretrain Ratio. We evaluate the performance of the original CDMs on the EDMCup2023 dataset with FACD integration at different pretrain ratios pt = {0.0, 0.1, 0.2, 0.3}, which represent the portion of the dataset used for pretraining before the CAT process. As shown in Figure 4, the original CDMs show performance drops with smaller pretrain ratios and require more steps in the CAT process. However, after integrating FACD, models quickly improve their performance in subsequent question selection steps, helping CDMs better align with students' real mastery levels.

NCD, Step = 1 NCD, Step = 5 NCD, Step = 10

Figure 5: t-SNE scatter plots for the distribution shift of students' Mas on NCD and FA-NCD.

The Distribution Shift of Students' Mas. Students can naturally be grouped based on their performance, such as high or low correct response rates, reflecting differences in their mastery level. To visualize this, we use t-SNE [Van der Maaten and Hinton, 2008] to project Mas onto a twodimensional plane (the 2D axes in t-SNE are for visualization only and hold no specific meaning), shading the scatter plot based on correct rates, where darker shades indicate higher correct rates. In Figure 5, the student representations learned by NCD with FACD on the NeurIPS2020 dataset exhibit clear clustering by the fifth step, becoming more distinct by the tenth step. In contrast, the standard NCD model shows no clear clustering at the fifth step, only displaying faint groupings by the tenth step. This demonstrates FACD's rapid adaptability in both representation learning and interpretability. Besides the visualization to show our model's interpretability, we also provide further comparison on the quantifiable interpretability metrics, such as DOA, in Appendix D.4.

5.5 Hyperparameter Analysis

We study the impact of the hyperparameters on the dynamic graph layer, GRU Layer, embedding dimension and graph mask ratio on the EDMcup2023 dataset. Specially, the model achieves optimal performance when the number of graph layers is set at 1 and 2, the number of GRU layers is set to 3, the embedding dimension is set to 32 or 64 and the graph mask ratio is set to 0.1 and 0.2. The detailed hyperparameter analysis and figure is provided in Appendix D.6.

6 Conclusion

In this paper, we propose a fast adaptive cognitive diagnosis framework called FACD for computerized adaptive testing. FACD focuses mainly on addressing the slow improvement issue of CDM in the early stage of CAT. Two dynamic diagnosis modules are introduced to capture collaborative information from similar students and critical information from individual response sequences, achieving fast adaptability. FACD still has some limitations, e.g., a lack of sufficient theoretical analysis to ensure effective alignment with CAT question selection strategies. Thus, future research could focus on strengthening the theoretical basis to enhance the interaction between CD and CAT.

Ethical Statement

The research presented does not involve human subjects or raise concerns related to privacy, security, or legal compliance. The datasets used in this study are from publicly available datasets that have undergone rigorous ethical review and anonymization during their release. These datasets contain no private student information and are widely used in various educational research studies.

Acknowledgments

The authors would like to thank the anonymous reviewers for their constructive and helpful comments. This work is supported by the National Natural Science Foundation of China (No. 62476091) and the National Undergraduate Training Program on Innovation and Entrepreneurship Grant (No. 202510269105G).