Denoised Attention and Question-Augmented Representations for Knowledge Tracing

1 Introduction

Knowledge tracing (KT) is an important task in online education systems, which focuses on predicting students' future performance based on their past learning interactions. This is accomplished by modeling students' knowledge states as they engage with learning platforms like massive open online courses and intelligent tutoring systems [Piech et al., 2015; Ghosh et al., 2020; Shen et al., 2021; Yin et al., 2023; Gao et al., 2025b]. Effectively addressing the KT task can help teachers better identify students who need further attention or recommend personalized learning materials to students. This is crucial to promoting the development of personalized education [Li et al., 2020; Huang et al., 2023].

∗Corresponding author.

Currently, mainstream KT models can be broadly divided into two major categories: deep sequence KT models [Medsker et al., 2001] and attention-based KT models [Vaswani et al., 2017]. The former (e.g., DKT [Piech et al., 2015], DKT-F [Nagatani et al., 2019], KQN [Lee and Yeung, 2019]) utilizes auto-regressive architectures, such as long short-term memory (LSTM) and gated recurrent unit (GRU), to capture students' interaction information at discrete timestamps, modeling temporal dependencies through iterative updates of hidden states. The latter (e.g., SAKT [Pandey and Karypis, 2019], SAINT [Choi et al., 2020], AKT [Ghosh et al., 2020]) employs multi-head attention mechanisms to flexibly model long-term dependencies in students' historical interaction sequences, often resulting in improved predictive accuracy. Compared to deep sequence models, attention-based models not only capture long-range dependencies more effectively but offer the advantage of parallel training. As a result, they have gradually become the dominant KT architecture in recent years.

Although attention-based KT models have shown promising results, they face the "attention noise" problem in practical applications: the model assigns non-negligible attention weights to some information that is essentially cognitively irrelevant, thereby generating interference signals. As shown in Figure 1, for the adjacent question, although their knowledge components (KCs) 1 have no intersection, the attention mechanism may mistakenly overestimate the correlation between them and assign them non-negligible attention weights, resulting in some questions that do have cognitive connections being ignored.

1 A knowledge component (KC) is a generalization of everyday terms like concept, principle, fact, or skill.

To effectively model students' forgetting behavior during cognitive processes, current mainstream attention-based KT models [Ghosh et al., 2020; Im et al., 2023; Yin et al., 2023; Li et al., 2024c] often introduce a decay mechanism that penalizes attention weights based on the temporal distance between student interaction logs-the greater the distance, the stronger the penalty-to simulate the effect of forgetting. However, the decay mechanism, to some extent, introduces noise into the attention. The decay mechanism included in the model emphasizes immediate learning behaviors, which may unintentionally lead to the neglect of learning materials with long-term cognitive significance [Salthouse et al., 2006]. This is similar to the theory of "irrelevant information inhibition" in cognitive psychology, which holds that in the cognitive process, individuals need to actively suppress irrelevant information and focus on task-related signals [McNamara and McDaniel, 2004; Li et al., 2023]. In KT task, if a model places too much emphasis on short-term learning behaviors while overlooking long-term cognitive development, it may struggle to accurately reflect students' evolving knowledge states. This imbalance can make it difficult to recognize students' gradual cognitive progress, thereby reducing the model's predictive accuracy. In addition, inaccurate input representation may also lead to attention noise problem. In educational scenarios, student's learning behaviors and the relationships between them are very complex. If the input of the model cannot accurately reflect these learning behaviors and the relationships between them, the attention mechanism cannot correctly focus on key information, resulting in attention noise.

To address the above attention noise problem, we present a novel KT model, called DenoiseKT, which focuses on the following two aspects. On the one hand, we enhance the representation by using a graph neural network to capture the complex relationships between questions, thereby obtaining a refined initial embedding representations. In addition, we also effectively model the difficulty of the questions. By combining the refined initial embedding representation and the modeling of the difficulty of the questions, we can finely encode the interaction sequence to reduce the interference of irrelevant information on the model. On the other hand, we refine the attention mechanism by introducing a weighting factor that suppresses the allocation of attention to irrelevant information. Through these two improvements, our model can solve the problem of attention noise, thereby improving the prediction performance of the model.

The main contributions of this paper are summarized as follows:

• We present the first solution to the attention noise problem in attention-based KT models, emphasizing the impact of irrelevant information on the distribution of attention weights.
• We propose the question-enhanced Rasch module (QERM) that can effectively capture the complex relationship between questions and effectively model the difficulty of questions, thereby making the feature representation more refined.
• We propose a denoised attention mechanism that can effectively reduce the model's attention weight allocation to irrelevant information.
• We conduct comprehensive quantitative and qualitative experiments to validate DenoiseKT on four public datasets, demonstrating significant performance improvements over existing 22 KT models and its effectiveness in alleviating the attention noise problem.

2 RelatedWork

2.1 Knowledge Tracing

KT aims to utilize historical learning interaction data from students to predict their responses to future questions. Recently, with the advancement of deep learning, this technology has been widely applied to KT tasks. We classify existing methods into deep sequence models and attention-based models based on the characteristics of KT models.

Deep sequence models employ auto-regressive architectures to capture the chronological order of student interactions. DKT [Piech et al., 2015] introduced LSTM layers to assess students' knowledge mastery over time, and KQN [Lee and Yeung, 2019] built upon DKT by incorporating a skill encoder, which combines both student learning behaviors and KCs representations to further refine the predictions made by DKT. Models like DKVMN [Zhang et al., 2017] leveraged memory modules for dynamic KT, and ATKT [Guo et al., 2021] incorporated adversarial techniques to create perturbations that enhance the generalization ability of the model. Other extensions, such as DIMKT [Shen et al., 2022] utilized multi-dimensional features to enhance predictions. However, deep sequence models often struggle with vanishing gradients and have a limited ability to capture long-range dependencies.

Attention-based models overcome many limitations of deep sequence models by utilizing self-attention mechanisms to capture long-term dependencies. SAKT [Pandey and Karypis, 2019] employed a self-attention network to model the relevance between KCs and students' past interactions, while SAINT [Choi et al., 2020] introduced an encoderdecoder architecture to represent sequences of exercise and response embeddings. AKT [Ghosh et al., 2020] implemented three self-attention modules, using a monotonic attention mechanism to explicitly model the phenomenon of student forgetting over time, and hybrid approaches such as SKVMN [Abdelrahman and Wang, 2019] and DTransformer [Yin et al., 2023] combined attention with additional features to enhance accuracy. These attention-based models excel in parallelized training and long-range dependency awareness, positioning them as the leading choice in modern KT applications.

Despite significant advancements in attention-based KT models, they still face some limitations: inaccurate input representation and excessive student forgetting modeling. These limitations often lead to the attention noise problem, where the model assigns non-negligible attention weight to some information that is cognitively irrelevant in nature, thereby generating interference signals.

2.2 Attention Noise

Attention noise refers to the phenomenon that the model based on the attention mechanism assigns non-negligible attention weights to segments of the input sequence that are either irrelevant or less important to the task at hand [Gao et al., 2025a]. As a result, the model fails to concentrate on the most crucial information in the sequence, instead incorporating irrelevant information into the decision-making process, which reduces the model's ability to capture the key patterns needed for accurate predictions or task execution, ultimately leading to a decline in model performance.

Attention noise has been widely recognized as a key problem affecting model performance. In recent years, many studies have tried to solve this problem from different angles. For example, [Zhao et al., 2019] indirectly alleviates attention noise by designing sparse attention patterns to focus on relevant parts. [Ye et al., 2024] proposed a differential attention mechanism to eliminate attention noise by calculating the attention score as the difference between two independent softmax attention maps.

However, these methods are not applicable to KT tasks, as attention noise is influenced by the modeling of student forgetting, often resulting in the noise being assigned more weight than the relevant information.

3 The Framework of DenoiseKT

3.1 Problem Statement

Given an arbitrary question q∗, KT's objective is to predict the probability of a student answering q∗ correctly based on the student's previous interaction data. More specifically, the previous interaction sequence of each student is represented as S = {< q1, C1, r1 >,< q2, C2, r2 >, ...,< qt, Ct, rt > }, where qt denotes the question answered by the student at time t, Ct represents the associated KC set to qt and rt is the binary response indicating whether the student's response to the question is correct (rt = 1) or incorrect (rt = 0) . We would like to estimate the probability of the student answering the question qt+1 correctly.

3.2 Framework Overview

DenoiseKT's overall framework is depicted in Figure 2. In this section, we provide a detailed introduction to DenoiseKT, which consists of three components: (1) interaction representation module that uses graph-enhanced question representation and incorporates question difficulty into the representation; (2) denoised attention module that effectively solves the problem of attention noise to better extract knowledge state from students' past learning history; (3) prediction module that uses a two-layer fully connected network to make prediction.

3.3 Interaction RepresentationModule

Effectively representing questions in interaction sequences is crucial to the success of the KT model. However, there are often complex and multi-level relationships between questions, making it difficult to comprehensively capture and model them using traditional methods alone. These relationships are not merely superficial adjacent connections (some questions may involve the same KC, thereby forming direct associations). Furthermore, there may exist deeper high-order relationships. For example, while question 1 and question 2 share a common KC, and question 2 and question 3 also share a KC, even though question 1 and question 3 lack direct KC connections, there may still exist inherent connections between them through this indirect chain of associations. Traditional models typically only handle the direct associations of these questions and struggle to capture more complex high-order relationships, which have a significant impact on the representation of questions.

To address the above problem, we utilize a graph convolutional network (GCN) to capture high-order relationships among questions to enhance question representations [Liu et al., 2024; Sun et al., 2023; Li et al., 2024a; Li et al., 2024b].

Given a set of questions Q and KCs C, we construct a question-KC bipartite graph G = (V, U), where V = (Q∪ C) and U = {(q, c)|q ∈ Q, c ∈ C} denote the sets of nodes and edges, respectively. We then use GCN combined with the question-KC bipartite graph to enhance question representations. The latent representation eqt of question qt is computed as follows:

where A is the adjacency matrix of G. Wq ∈ Rd×nq is the embedding matrix of question. o∗ is the original one-hot vector of ∗. W ∈ Rd×d and b ∈ Rd are learnable linear transformation operations.

Additionally, even for questions that cover the same KC, there may be significant differences in difficulty. For instance, two questions that both assess fraction addition: one is a simple addition with the same denominator, while the other involves more complex operations with different denominators, students' performance on these two questions will demonstrate a significant difference. If we judge students' performance solely based on whether they have mastered a certain type of question, while ignoring the difficulty of the question, the model may make incorrect inferences. Inspired by the classic Rasch model [Rasch, 1993], which models individual ability and item difficulty separately, we incorporate question difficulty into the question representation. After obtaining the latent representation eqt of question qt, we enhance it by introducing a difficulty parameter to better capture the underlying properties of the question. More specifically, the t-th representations of question and interaction are represented as follows:

where dqt represents the question qt difficulty. ect and ert denote the latent representations of KC set Ct and student response rt on question qt. |Ct| is the total number of KCs associated with question qt. Wd ∈ Rd×nq , Wc ∈ Rd×nc and Wr ∈ Rd×2 represent the difficulty embedding matrix, the KC embedding matrix, and the response embedding matrix. ⊕ and ⊙ represent the addition operators and element-wise product respectively.

The above is a detailed overview of the QERM we proposed. This module can not only characterize the complex relationships between questions, but also reasonably model the difficulty differences of questions so as to represent questions more accurately.

3.4 Denoised AttentionMechanism

In the research field of KT and educational measurement, students' cognitive behaviors are closely linked to the questions they answer. When answering questions, students tend to focus on those questions that are most similar in content and closest in time to the current question, and answer based on this information. This process conforms to the laws of human cognition, because the human brain typically processes information through associations, especially when confronted with new problems, it automatically recalls similar situations and experiences.

In real cognitive processes, the factors that have the most significant impact on a student's current cognitive behavior are often those historical questions that are most relevant and closest to the current question. However, existing attentionbased models KT models tend to assign non-negligible attention scores to questions that are not closely related to the current question, due to inaccurate input representations and the decay mechanisms introduced to account for student forgetting. These results in unnecessary interference and prevents the model from focusing on truly relevant information. These non-negligible irrelevant attention scores are referred to as attention noise. This noise affects the model's ability to accurately track and analyze the student's learning process, thereby reducing the model's overall effectiveness. To solve the attention noise problem, we design a denoised attention mechanism that multiplies the attention scores by weight factor. This enables the model to effectively focus on truly relevant information. Specifically, the retrieved knowledge state ht+1 at the (t + 1)-th timestamp is determined using the following formula:

where L(·), KT and d denote the linear operation, the transpose of K and the dimension of K respectively. BF is the weight factor that is calculated based on the similarity and distance between questions in the interaction history. Specifically, each element of BF is computed as follows:

where bfij denotes the element at the (i)-th row and (j)-th column of BF. 1{qi≃qj} means it is 1 when qi and qj share the same KC, and 0 otherwise. β is a hyperparameter greater than 0 and less than 1.

3.5 PredictionModule

To predict the outcomes of students over questions, we employ a two-layer fully connected neural network following the attention layers. The attention layers capture a comprehensive representation of the student's behavioral sequence. After passing through the feed-forward network in the attention module, this representation, denoted as ht+1, functions as a summary of the knowledge state. This is subsequently merged with the contextualized embedding of the current question xt+1, creating the input for the prediction layer. The prediction layer calculates the probability of a correct response ˆrt+1 using the following formulation:

where σ(·), η(·) denote Sigmoid and ReLU functions. W1 ∈ Rd×2d , W2 ∈ Rd×d , b1 ∈ Rd , b2 ∈ Rd are learnable parameters.

We optimize the prediction function by minimizing the binary cross-entropy loss between the ground-truth response rt+1 and the prediction probability ˆrt+1 as follows:

This predictive framework allows the model to incorporate information from both previous interactions and current contexts, ensuring precise estimates of students' knowledge states and response probabilities.

4 Experiment

To evaluate the performance of our proposed DenoiseKT framework, we conduct extensive experiments to address the following research questions:

• RQ1: Can DenoiseKT outperform existing KT models in terms of prediction performance?
• RQ2: How does each component of DenoiseKT contribute to its overall effectiveness?
• RQ3: Can DenoiseKT effectively solve the attention noise problem?

4.1 Datasets

In this paper, we evaluated the performance of our model on four widely used publicly available datasets.

• ASSISTments2009 (ASSIST2009) 2 : This dataset is collected from the ASSISTment online tutoring platform during the 2009-2010 school year and focuses on math exercises. It has been a standard benchmark for KT methods over the past decade. The dataset includes 346,860 interactions, 4,217 students, 17,737 questions, and 123 KCs, with the maximum number of KCs in a single question being 4.
• NeurIPS2020 Education Challenge (NIPS34) 3 : This dataset is provided by NeurIPS 2020 Education Challenge. It contains students' responses to mathematics questions from Eedi. The dataset includes data from Task 3 and Task 4, with 1,399,470 interactions, 4,918 students, 948 questions, 57 KCs, and the maximum number of KCs in a single question is 2.
• XES3G5M 4 : This large-scale dataset derived from a Chinese online mathematics learning platform, documenting the learning performance of third-grade students in mathematics. The dataset contains 5,549,635 interactions involving 18,066 students across 7,652 math questions. It includes rich auxiliary information,such as 865 KCs with hierarchical relationships, question types,

2 https://sites.google.com/site/assistmentsdata/home/2009-2010assistment-data/skill-builder-data-2009-2010

3 https://eedi.com/projects/neurips-education-challenge

4 https://github.com/ai4ed/XES3G5M

textual content and analysis, as well as timestamps of student responses. In terms of the number of KCs and the richness of contextual information, this dataset is currently the largest in the mathematics domain.

• EdNet 5 : This dataset is collected over two years by Santa, an AI tutoring service with over 780,000 users in Korea. Given the large volume, we randomly select a subset of student records for model evaluation. The dataset includes 597,042 interactions, 4,999 students, 11,901 questions, and 188 KCs, with each question associated with up to 7 KCs.

To ensure reproducibility in our experiments, we rigorously follow the data pre-processing steps suggested in [Liu et al., 2022b]. We filter out student sequences that are shorter than 3 interactions. Data statistics are summarized in Table 1.

4.2 Baselines

To assess the performance of DenoiseKT, we compared it with 22 state-of-the-art KT models as follows: DKT [Piech et al., 2015], DKT+ [Yeung and Yeung, 2018], DKTF [Nagatani et al., 2019], KQN [Lee and Yeung, 2019], DKVMN [Zhang et al., 2017], ATKT [Guo et al., 2021], GKT [Nakagawa et al., 2019], SAKT [Pandey and Karypis, 2019], SAINT [Choi et al., 2020], AKT [Ghosh et al., 2020], SKVMN [Abdelrahman and Wang, 2019], HawkesKT [Wang et al., 2021], Deep-IRT [Yeung, 2019], DIMKT [Shen et al., 2022], qDKT [Sonkar et al., 2020], AT-DKT [Liu et al., 2023], simpleKT [Liu et al., 2022a], QIKT [Chen et al., 2023], RKT [Pandey and Srivastava, 2020], FoLiBiKT [Im et al., 2023], DTransformer [Yin et al., 2023], extraKT [Li et al., 2024c].

It is important to note that the baseline models selected in this paper are mostly derived from articles published in top venues on AI/ML or education. Furthermore, when comparing these baseline models, the implementations were based on the authors' publicly available code and reproduced following the configuration settings in pyKT [Liu et al., 2022b]. All methods underwent optimal hyperparameter tuning using Weights&Bias 6 , ensuring the fairness of the comparison results. Moreover, the hyper-parameter tuning tool Weights&Biases utilizes Bayesian search to automatically identify the optimal parameter combinations. For each model, the tool explores approximately hundreds of combinations during the tuning process. This automated approach ensures that all methods achieve optimal parameters on each dataset.

5 https://github.com/riiid/ednet

6 https://wandb.ai/site/

4.3 Experimental Setting

In order to fairly evaluate the performance of KT models, all models are trained and evaluated on student interaction sequences of fixed length 200. For each dataset, we use 20% of all student sequences for the test set, and perform standard 5fold cross validation on the rest 80% of all student sequences. We use the Adam optimizer to train our model. The maximum of the training epochs is set to 200, and we choose to use early stopping when the performance is not improved after 10 epochs. The embedding dimension, the hidden state dimension, the two dimension of the feed-forward neural network layers are set to [64, 128], the learning rate, random seed, dropout rate and hyper-parameter β are set to [1e-3, 2e3, 1e-4], [42, 3407], [0.1, 0.2, 0.3, 0.4, 0.5] and [0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99] respectively, the number of blocks and attention heads are set to [1, 2, 4] and [4, 8]. Our model is implemented in PyTorch [Imambi et al., 2021] and trained on a cluster of Linux servers with the NVIDIA RTX 3090 GPU device. Aligned with previous work [Liu et al., 2022b; Ghosh et al., 2020; Liu et al., 2022a], we use area under the curve (AUC) as the main evaluation metric and accuracy (ACC) as the secondary evaluation metric.

4.4 Results

Overall Performance (RQ1)

Table 2 shows the overall performance of all models in the four datasets. We calculate the mean and standard deviation of the AUC and ACC across 5 folds. According to Table 2, we have the following observations: (1) on the main evaluation metrics, our proposed model DenoiseKT outperforms all state-of-the-art models. Compared to the second-best performing model, the DenoiseKT model performs 0.20% better on ASSIST2009, 0.01% better on NIPS34, 0.57% better on XES3G5M, and 0.82% better on EdNet. On the secondary evaluation metrics, our proposed model DenoiseKT outperforms almost all baselines (except QIKT and extraKT on the NIPS34 dataset) , with scores of 0.7427 (ASSIST2009), 0.7330 (NIPS34), 0.8319 (XES3G5M), and 0.7131 (EdNet). These results indicate the effectiveness of DenoiseKT; (2) compared to all attention-based KT models, i.e., SAKT, SAINT, AKT and simpleKT, our model has the best performance on all four datasets. This indicates our denoised attention mechanism and QERM allow attention-based KT models to effectively deal with the attention noise problem that improves the predictive performance.

Although the AUC improvements shown in Table 2 may appear modest, with increases of less than 1% on some datasets, these gains are significant, especially when evaluated with a context window size of 200. Recent benchmarking studies have highlighted that many reported advancements in KT models are unreliable, often stemming from flawed evaluation methods, with overall prediction performance having improved by only 3.5% since 2015. In contrast, our study strictly adhered to the rigorous evaluation protocol proposed by pyKT [Liu et al., 2022b] and conducted an exhaustive hyper-parameter search for all baseline models to ensure reliable and meaningful comparisons.

Ablation Study (RQ2)

In order to verify the effectiveness of each design component in the DenoiseKT model, we constructed three variants of the DenoiseKT model for ablation experiments, as shown in Figures 3 and 4. Specifically, for variant DenoiseKT w/o QERM, we removed the QERM in the question representation part, that is to say we no longer use GCN to capture the complex relationship between questions to enhance question representation and also cancel the effective modeling of problem difficulty. We use the standard method commonly used in KT research to represent the question. For variant DenoiseKT w/o BF, we removed the weight factor BF used to solve the noise problem in the denoised attention mechanism. For variant DenoiseKT w/o QERM & BF, we removed both the QERM and the weight factor BF.

It can be observed that compared with the complete DenoiseKT model, all other ablation variants perform worse on the four datasets. Specifically, (1) the variant without QERM (DenoiseKT w/o QERM) had different degrees of performance degradation on all datasets, among which the performance of ASSIST2009 was the most obvious, with AUC down by 1.3%. This emphasizes the importance of capturing the complex relationship between problems and effectively modeling the difficulty of problems; (2) the variant without BF (DenoiseKT w/o BF) also leads to a decrease in performance, which shows that reducing the generation of attention noise is crucial for accurate prediction; (3) compared with other variants, the variant without QERM and BF (DenoiseKT w/o QERM & BF) shows the worst results.

Overall, the absence of any component will lead to a decrease in model performance, which shows that each module in the DenoiseKT model plays an important role in the KT task.

Visualization (RQ3)

To intuitively demonstrate the difference between regular dotproduct attention and denoised attention, we visualize the attention scores of both regular dot-product attention and denoised attention, as shown in Figure 5. From Figure 5, we have the following observations: our denoised attention can capture the information of similar problems well, while regular dot product attention cannot capture the information of similar problems well. For example, when predicting the 9th question, denoised attention focuses on information from questions 4, 3, 2, and 1, while regular dot-product attention emphasizes information from questions 8, 7, 6 and 5. This demonstrates that our model can efficiently tackle the issue of attention noise.

5 Conclusion

This paper presents DenoiseKT, a KT model designed to address attention noise by introducing two key components: a QERM for refined question representation, and a denoised attention mechanism to suppress irrelevant signals. Experiments on four public datasets confirm that both components contribute to improved prediction accuracy. Beyond performance gains, DenoiseKT offers a generalizable and cognitively grounded framework for calibrating attention in educational modeling, paving the way for more interpretable and reliable KT systems.

Acknowledgements

This work was supported in part by National Key R&D Program of China, under Grant No. 2023YFC3341200; in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (2022LSYS003) and in part by Beijing Municipal Science and Technology Project under Grant No. Z241100001324011.