DGCPL: Dual Graph Distillation for Concept Prerequisite Relation Learning

1 Introduction

As the increasing of learners number on online learning platforms, the learning resources also become more abundant [O'Dea and Stern, 2022]. However, the diversity of resources often leads to difficulties in choice for learners. The main reason for this is that the dependency between the core concepts in different learning resources is unclear. Concept prerequisite relations are the type of knowledge dependency, which indicates that the learning order of core concepts in a domain determines the learning order of knowledge. For example, in mathematics, understanding the concept of "Markov chain" requires prior knowledge of "Probability" and "Stochastic process". Thus, Concept prerequisite relations play an indispensable role in various educational scenarios, such as learning path planning [Yu et al., 2024; Wang and Liu, 2016], knowledge tracing [Huang et al., 2024; Yin et al., 2023], cognitive diagnosis [Song et al., 2023; Gao et al., 2023], and learning resource recommendation [Dong et al., 2024; Wang et al., 2022a].

∗Corresponding author.

Early research with machine learning for concept prerequisite relations [Talukdar and Cohen, 2012; Liang et al., 2015] primarily rely on manual feature extraction of concept pairs for concept modeling. This involves designing a set of metrics to capture the semantic and structural information between concepts, but the process is both tedious and time-consuming. With the rise of graph neural networks, researchers begin to explore the intrinsic relations between knowledge concepts in graphs [Sun et al., 2022; Zhang et al., 2022a]. However, previous research mostly infer prerequisite relations between knowledge concepts from the knowledge perspective [Zhang et al., 2025]. Few research approach the problem from the perspective of learning behavior. In fact, if many learners, after studying concept B, immediately look up the definition of concept A, there is likely a prerequisite relation between these two concepts. Existing research on learning behaviors [Sayyadiharikandeh et al., 2019; Hu et al., 2021] predominantly adopts traditional machine learning methods. While these studies recognize the importance of learning behaviors, they have yet to explore advanced deep learning methods for deeper insights.

However, current graph-based methods usually only focus on pairwise relations of concept-concept or conceptresource. They do not address the set-level relations between a learning resource and multiple knowledge concepts, which may lead to the omission of key information. On the other hand, few research [Sayyadiharikandeh et al., 2019; Hu et al., 2021] use learning behavior features to predict prerequisite relations, but these still rely on traditional machine learning methods. How to combine advanced deep learning methods and predict prerequisite relations from both the knowledge and learning behavior perspectives is an urgent research question, as shown in Figure 1.

To this end, we propose a Dual Graph Distillation method for Concept Prerequisite Relation Learning (DGCPL), which aims to construct a dual graph model from both the knowledge and learning behavior perspectives to capture concept prerequisite relations. Specifically, the Concept-Resource Hypergraph represents the higher-order knowledge relations between concepts and resources. The Learning Behavior Graph describes the learner's behavior on knowledge concepts. To effectively integrate information from different perspectives, we design gated knowledge distillation to combine the knowledge structure features of concepts and dynamic behavior features of learners, and adaptively adjust and optimize two types of information to identify prerequisite relations.

The main contributions of this paper are as follows:

• We propose a Dual Graph Distillation method for concept prerequisite relation learning, which predicts concept prerequisite relations by constructing a dual graph from the knowledge and learning behavior perspectives.
• We introduce gated knowledge distillation that adaptively integrates higher-order knowledge relations and learning behavior features information in the dual graph through a gating mechanism, resulting in more comprehensive representations of concept embeddings.
• On three publicly available benchmark datasets, we compare DGCPL with eight graph-based baseline methods and five traditional classification baseline methods. The experimental results demonstrate that DGCPL achieves state-of-the-art performance.

2 RelatedWork

2.1 Concept Prerequisite Relation Learning

Talukdar and William pioneer the task of extracting concept prerequisite relations from Wikipedia [Talukdar and Cohen, 2012], laying the foundation for subsequent research. Early machine learning methods, such as RefD [Liang et al., 2015], propose a metric based on citation distance; Sayyadiharikandeh et al. are the first to analyze learning behavior using Wikipedia Clickstream data [Sayyadiharikandeh et al., 2019]; and PREREQ [Roy et al., 2019] employs siamese networks to identify prerequisite relations between concepts. These research further advance the domain. In the context of graph neural networks, research gradually shifts to graph modeling. ConLearn [Sun et al., 2022] combines concept graphs with self-attention mechanisms to learn prerequisite relations in a context-aware manner. MHAVGAE [Zhang et al., 2022a], HGAPNet [Mazumder et al., 2023], and LCPRE [Sun et al., 2024] extract prerequisite relations by constructing heterogeneous concept-resource graphs, offering new insights for modeling complex concept relations.

2.2 Knowledge Distillation

Knowledge Distillation [Hinton et al., 2015] is a model compression technique, which aims to transfer the knowledge of a complex teacher model to a lightweight student model. To integrate diverse data more effectively, the DGEKT model [Cui et al., 2024] in the knowledge tracing research field incorporates a gating mechanism into online knowledge distillation to dynamically adjust the knowledge transferred from the teacher model to the student model. Traditional knowledge distillation achieves knowledge transfer by minimizing the difference between the outputs of the teacher and student models. However, many subsequent studies [Romero et al., 2015; Zagoruyko and Komodakis, 2017; Pech´ac et al., 2024] have found that extracting representation features from the intermediate layers of the teacher model can further improve the effectiveness of distillation. Additionally, the introduction of online knowledge distillation [Yang et al., 2023; Gong et al., 2023] and self-distillation [Zhang et al., 2022b; Li et al., 2024] has opened new directions for the development of knowledge distillation.

Remarks. We construct a dual graph structure from the perspectives of knowledge and learning behavior, using hypergraph and directed graph to model the concept-resource hypergraph and the learning behavior graph, respectively. Then we use gated knowledge distillation to integrate high-order knowledge relations between concepts and resources with learning behavior features.

3 Problem Statement

In this section, we present the key terms and research definition used in this paper. Let C = {c1, c2, c3, ..., cm} represent the set of knowledge concepts, and R = {r1, r2, r3, ..., rn} represent the set of learning resources, where m and n denote the total number of knowledge concepts and learning resources, respectively. A knowledge concept refers to a core knowledge unit in a specific domain. And a learning resource refers to an independent learning object, typically a chapter in a textbook or the subtitles of a course video.

Concept-Resource Hypergraph. In the hypergraphGH = (V, EH), nodes represent concepts, and hyperedges represent resources. Here, V = {v1, v2, ..., vm} denotes the set of nodes, and EH = {h1, h2, ..., hn} denotes the set of hyperedges. The relation between hyperedges and nodes represents the containment relation between resources and concepts.

Learning Behavior Graph. In the directed graph GL = (V,EL), nodes represent concepts, and edges represent the navigation behavior of learners between concepts. EL denotes the set of edges in the graph, where lij ∈ EL represents the number of times learners navigate from the definition document of concept ci to the definition document of concept cj within a certain time range.

Here, the concept definition documents are not the same as the learning resources mentioned above. It is often challenging to directly obtain clickstream data from learning resources. Therefore, We uses the Wikipedia page articles corresponding to concepts as their definition documents to collect learning behavior data and construct the learning behavior graph. Clickstream data only includes concept pairs with more than 10 navigation instances in Wikipedia. Clearly, this clickstream data represents learning learning behavior with distinct group characteristics, which better indicates the potential relation between two concepts. Such dynamic learning behaviors can reflect the concept dependency relations.

Research Definition. Given a set of knowledge concepts C and a set of learning resources R within a specific domain, our goal is to learn a function Fθ : C ×C → {0, 1} that can predict whether any pair of concepts < ci, cj > in the domain has a prerequisite relation.

4 Methodology

In this paper, we introduce an innovative model designed to learn concept prerequisite relations using a dual graph distillation strategy. The structure of DGCPL consists of three main modules: dual graph construction, gated knowledge distillation, and ensemble prediction network. Figure 2 illustrates the overall architecture of our proposed DGCPL.

4.1 Dual Graph Construction

Concept-Resource Hypergraph

We adopt a hypergraph to capture the high-order knowledge relations between knowledge concepts and learning resources, thereby learning concept embeddings within the hypergraph. Specifically, we first construct hypergraph GH = (V,EH). The hypergraph adjacency matrix is denoted as H ∈ Rn×m. If a node vi belongs to a hyperedge hi, Hi,hi = 1; otherwise, Hi,hi = 0. Additionally, each hyperedge is assigned a weight wh, and all these weights are stored in a diagonal matrix W ∈ Rm×m. For simplicity, we set the weights of all hyperedges to 1. We update the embeddings of concept nodes using a Hypergraph Convolutional Network (HGCN) [Feng et al., 2019],

Here, x(l) i ∈ RCl represents the embedding of the i-th node at the l-th layer. D ∈ Rn×n is the degree matrix of nodes, where the degree of node vi is Di,i = ∑mi=1 whi Hi,hi . B ∈ Rm×m is the degree matrix of hyperedges, where the degree of hyperedge hi is Bhi,hi = ∑ni=1 Hi,hi . Θ(l+1) ∈ RCl×Cl+1 is the learnable weight matrix at the (l + 1)-th layer. ϕ denotes the ReLU activation function, and (·)T denotes the transpose operation. To retain the original critical information, we add x(l) as a residual and apply a linear transformation linear(·). The initial embedding x(0) i ∈ RC0 of each concept is generated by inputting the concept definition document into a pretrained language model.

Finally, we obtain the concept embedding xHi ∈ Rd from the concept-resource hypergraph, where d represents the dimension of the concept embeddings.

Learning Behavior Graph

In addition to the high-order knowledge relations between learning resources and knowledge concepts, learners' behaviors also have a significant impact on predicting concept prerequisite relations. This has been demonstrated in previous studies [Sayyadiharikandeh et al., 2019; Hu et al., 2021]. Therefore, we construct a learning behavior graph based on learning behaviors to predict concept prerequisite relations. Specifically, for a node vi in the learning behavior graph GL = (V,EL), we let nj,i represent the number of times learners navigate from node vj to node vi. The in-degree of node vi with respect to node vj is defined as:

where V represents the set of all knowledge concepts in the domain. Similarly, let ni,j represent the number of times learners navigate from node vi to node vj. The out-degree of node vi with respect to node vj is defined as:

In addition, we set A(in) i,i = 1 and A(out) i,i = 1. Then, we define the in-degree and out-degree of node vi as follows:

In the learning behavior graph, each edge has a direction, which makes information propagation more complex. Inspired by previous research [Cui et al., 2024], we utilize a Directed Graph Convolutional Network (DGCN) [Tong et al., 2020; Zhang et al., 2021] to learn concept embeddings from the learning behavior graph. Nodes can not only propagate information to other nodes but also receive information from other nodes. This parallel bidirectional information propagation mechanism enables more accurate capture of concept dependency relations based on learners' dynamic behaviors.

For the in-degree directed graph (denoted as "-") of the learning behavior graph, we use DGCN to propagate information along the in-degree edges. The embedding of node at the (l + 1)-th layer is calculated as follows:

where N(i)- represents the in-degree neighbor set of node vi (including node vi itself). Θ(l+1),- N is the learnable weight matrix at the (l + 1)-th layer in the in-degree directed graph. x(l) j represents the embedding of node j at the l-th layer. Similarly, the initial embedding of each concept, x(0) i ∈ RC0 is generated by inputting the concept definition document into a pretrained language model.

Similarly, to achieve information propagation along the out-degree edges, we also apply DGCN to the out-degree directed graph (denoted as "+"). The embedding of node vi at the (l + 1)-th layer is calculated as follows:

where N(i)+ represents the out-degree neighbor set of node vi (including node vi itself), and Θ(l+1),+ N is the learnable weight matrix at the (l + 1)-th layer in the out-degree directed graph. Then, for the entire learning behavior graph, we fuse the updated concept embeddings obtained from the in-degree directed graph and the out-degree directed graph. The embedding of node vi at the (l+1)-th layer is defined as:

Here, we apply a linear transformation to both embeddings separately and then add them element-wise. Additionally, there are various options for embedding fusion methods. Besides element-wise addition, common methods include embedding concatenation, element-wise maximum, and element-wise average. We provide a detailed experimental analysis of these embedding fusion methods in Appendix A. Finally, we obtain the concept embedding xLi ∈ Rd from the learning behavior graph.

4.2 Gated Knowledge Distillation

To effectively integrate features from the knowledge perspective and the learning behavior perspective, we apply gated knowledge distillation to adaptively integrate information from the dual graph, getting more comprehensive concept embedding representations. Specifically, we treat the concept-resource hypergraph and the learning behavior graph as two independent student models. Subsequently, we regard these two graph modules as equivalent student models and employ a gating mechanism to achieve their effective integration, thereby constructing a stronger teacher model. The gating mechanism [Hochreiter and Schmidhuber, 1997] is a method for controlling information flow. It adaptively adjusts the information flow weights between the concept-resource hypergraph and the learning behavior graph based on the final node embeddings generated by the two student models, optimizing the model's decision process.

The concept node embeddings xHi and xLi , generated from the concept-resource hypergraph and the learning behavior graph, respectively, contain both the textual semantic features and the graph topological structure features of the node. To construct the teacher model, we use a gating mechanism to integrate these two embeddings:

here,

When g is close to 1, xHi contributes more to the fused embedding; while when g is close to 0, xLi contributes more. σ represents the sigmoid activation function, and ⊙ denotes the hadamard product. Wf and bf are the learnable weight matrix and bias vector, respectively. Finally, we obtain the final embedding xTi ∈ Rd of the teacher model.

The teacher model is constructed from the student models through online knowledge distillation, where both models are updated during training. The teacher model generates predictions based on the student model's output and feeds them back as supervision signals. This enables the teacher model to integrate complex concept structures and dynamic learning behavior, helping the student models learn richer features.

4.3 Ensemble Prediction Network

Learning concept prerequisite relations is essentially a binary classification task. After obtaining the final concept representations, a Siamese network [Bromley et al., 1993] is used to predict whether concept ci is a prerequisite for concept cj. The embeddings of the two concepts are separately fed into two feedforward networks with shared weights:

Here, xi can be any of xHi , xLi , or xTi . W1 and b1 are the weight matrix and bias vector, respectively. The two outputs are then concatenated for classification, which can be expressed as follows:

where ∥ represents embedding concatenation, and Wp and bp are the parameterized weight matrix and bias vector, respectively. Through three Siamese networks with the same structure but different parameters, we can obtain the predicted probabilities of the two student models and the teacher model, denoted as pH (ci, cj), pL(ci, cj), and pT (ci, cj), respectively. For these three models, we use the binary cross-entropy loss function(BCE) to calculate the prediction loss:

where D represents the training dataset for concept prerequisite relations, and |D| is the size of the training set. ycicj ∈ {0, 1} indicates ground truth. We denote the prediction losses of the three models as LHpreq , LLpreq , and LTpreq , respectively. Additionally, we set the classification prediction threshold γ to 0.50. When the predicted probability is greater than or equal to γ, the concept pair < ci, cj > is considered to have a prerequisite relation; otherwise, it does not.

Next, to distill the knowledge of the integrated teacher model back to the student models, we first calculate the soft version of the predicted probabilities as temperature τ :

where the temperature coefficient τ is set to 0.5. We obtain the soft versions of the predicted probabilities for the two student models and the teacher model, denoted as ˆ pH (ci, cj ), ˆ pL(ci, cj), and ˆ pT (ci, cj), respectively. Then, we calculate the distillation loss to encourage each student model to align its predictions with the teacher model's predictions:

where ∥·∥1 represents the L1 norm. In this case, we use the L1 norm instead of KL divergence. Studies [Wang et al., 2022b] have shown that using the L1 norm as part of the loss function can more effectively constrain the model's prediction error, thereby improving the model's classification performance.

Finally, the overall loss function of the model is defined as:

where λ is a hyperparameter that balances the distillation loss.

5 Experiment

5.1 Experimental Setup

Datasets. To evaluate the effectiveness of our proposed model, we select three public benchmark datasets.

• University Course Dataset (UCD) 1 : This dataset compiles course information [Liang et al., 2017] from the computer science domain across 11 universities in the United States, covering various topics such as algorithm design, computer graphics, and neural networks.
• LectureBank 2 : This dataset [Li et al., 2019] originates from online education platforms, covering five domains: natural language processing, machine learning, artificial intelligence, deep learning, and information retrieval.
• MOOC 3 : This dataset [Liang et al., 2017] is derived from video playlists in the MOOC corpus and includes the subtitle texts of videos from 38 playlists in the computer science department.

Data Preprocessing. Since our model will utilize the user clickstream data fromWikipedia in 2019 as the learning behavior data for the learning behavior graph. In LectureBank, 74 concepts could not be matched to Wikipedia page names and were merged with others of the same meaning, so we removed them without affecting domain coverage. Additionally, we follow the method in previous research [Sun et al., 2024] to generate negative samples equal in number to the positive samples, enhancing the model's robustness. Specifically, half of the negative samples were created through unrelated random sampling, while the other half are reversed pairs of the original positive samples. The final statistics of the three experimental datasets are shown in Table 1.

Baseline Methods. We compare our DGCPL with eight graph-based methods for predicting concept prerequisite relations: LCPRE [Sun et al., 2024], HGAPNet [Mazumder et al., 2023], MHAVGAE [Zhang et al., 2022a], ConLearn [Sun et al., 2022], R-VGAE(T) [Li et al., 2020], R-VGAE(P) [Li et al., 2020], VGAE [Kipf and Welling, 2016], and GAE [Kipf and Welling, 2016]. Additionally, we considered five traditional classification baseline methods: PREREQ [Roy et al., 2019], RefD [Liang et al., 2015], Random Forest (RF), Support Vector Machine (SVM), and Naive Bayes (NB). Please refer to Appendix B for detailed descriptions.

Evaluation Metrics. For the fair and extensive evaluation, we select three common evaluation metrics to assess the performance of all methods: Accuracy (ACC), F1 Score (F1), and Area Under the ROC Curve (AUC).

1 https://github.com/suderoy/PREREQ-IAAI-19

2 https://github.com/Yale-LILY/LectureBank

3 https://github.com/suderoy/PREREQ-IAAI-19

Implementation Details. We split each datasets into training, validation, and test sets with a ratio of 8:1:1. Our proposed model is trained using the Adam optimizer for a total of 50 epochs. The learning rate lr, batch size, and classification prediction threshold γ are set to 1E-4, 16, and 0.50. For the UCD, LectureBank, and MOOC datasets, the number of graph neural network layers ℓ is set to 3, 2, and 2; the weight decay is set to 1E-4, 1E-2, and 1E-3; and the distillation loss weight λ in the overall loss function is set to 1E-6, 1E-1, and 1E-5, respectively. We use BERT [Devlin et al., 2019] as the pretrained language model. Detailed analysis of the impact of the pretrained language model is provided in Appendix C. All experiments is implemented on the Linux sever with one RTX 4090D GPU using the PyTorch framework.

5.2 Overall Performance

In Table 2, we present the comparative results of all methods on the three datasets. Our model generally outperforms existing baselines across all datasets. Although the AUC metric shows a slight decline on the LectureBank dataset, this minor difference may result from the complex concept relations or the imbalanced sample distribution in the dataset. However, DGCPL achieves significant improvements in AUC on the other two datasets, demonstrating its adaptability and robustness in handling different types of educational data.

In summary, we draw the following conclusions: (1) DGCPL significantly outperforms existing baseline models in all evaluation metrics on the three datasets, showcasing its strong capability in learning concept prerequisite relations. (2) Graph-based methods perform better than traditional methods because the relations between concepts, between concepts and resources, and between resources form a knowledge network, which provides a significant advantage. (3) The HGAPNet and LCPRE models perform as the next best methods. They fully leverage the complex interaction relations between concepts and resources, but lack the perspective of learning behavior.

5.3 Ablation Experiment

In this study, to evaluate the effectiveness of each module in DGCPL, we designed three variants and compared them with these variants in terms of F1. The ablation study results in Table 3 show that: (1) Removing the Gated Knowledge Distillation (GKD) leads to a slight performance decrease. This reduction in performance occurs because the model loses its ability to integrate knowledge across graphs. (2) Removing the Concept Resource Hypergraph (CRHG) along with the associated GKD significantly reduces performance on all datasets, highlighting the importance of concept embeddings from the knowledge perspective. (3) Removing the Learning Behavior Graph (LBG) along with the associated GKD results in a substantial performance drop. This group learning navigation behavior indeed reflects the prerequisite relations between concepts and deserves attention.

5.4 Quality Analysis

In a knowledge domain, the number of neighboring concepts (i.e., prerequisite concepts and successor concepts) varies across different concepts. To evaluate our model's performance under different numbers of neighbors, we conduct experiments on LectureBank, dividing concepts into lowdegree, medium-degree, and high-degree groups based on the trichotomies of neighbor count. The pie chart in Figure 3 shows the proportion of concepts in each group. We compare our method with three advanced baselines on the test set. From the bar chart in Figure 3, we observe: (1) DGCPL outperforms the baselines in predicting low-degree, medium-degree, and high-degree concepts, demonstrating robustness in predicting sparse prerequisite relations. (2) For each model, the prediction performance for low-degree concepts is consistently the lowest. Current graph-based methods tend to degrade in performance when knowledge relations are sparse, highlighting the importance of considering the learning behavior perspective. However, Our model also excels in predicting low-degree concepts.

5.5 Hyperparameter Analysis

As illustrated in Figure 4, we evaluate the performance of the model on UCD under different hyperparameter settings. The results are summarized as follows: (1) The model performs best when the graph neural network depth ℓ is 3, as lower depths fail to capture complex graph structures, and deeper networks may overfit or excessively smooth information. (2) The distillation loss weight λ has an optimal range, with performance improving as the weight decreases from 1E-3 to 1E-6, but declining when it drops further to 1E-7 due to insufficient learning from the teacher model. (3) The classification prediction threshold γ of 0.50 achieves optimal performance; lower thresholds lead to an increase in false positives, while higher thresholds reduce the model's ability to correctly identify positive samples. (4) The optimal learning rate lr is 1E-4, which ensures fast and stable convergence. The same method is applied to select the best hyperparameter settings for the other datasets.

5.6 Training Overhead Analysis

In this study, we compare the model parameter sizes and training times between DGCPL and three advanced baselines on MOOC. As illustrated in Figure 5, we draw the following conclusions: (1) Despite the increased parameter count due to the dual graph structure, DGCPL achieves faster learning of better concept embeddings. The dual graph structure captures concept features from both knowledge structure and learning behavior perspectives, and gated knowledge distillation accelerates convergence, reducing the number of epochs needed for training. (2) LCPRE incurs significant overhead due to depth-first search for exploring concept-resource paths. (3) HGAPNet updates all node and edge features, slowing convergence, and its small batch size further hinders speed. (4) MHAVGAE has faster convergence, but the number of attention heads affects its optimization speed.

6 Conclusion

This paper proposes a dual graph distillation method for learning concept prerequisite relations. The method aims to construct a dual graph structure, including the conceptresource hypergraph and the learning behavior graph, from the perspective of knowledge and the perspective of learning behavior. Then we effectively integrate the dual graph through gated knowledge distillation to construct a more robust teacher model to predict concept prerequisite relations. This method not only leverages the high-order knowledge relations between concepts and resources, but also integrates the learner's learning behavior features, thereby enhancing the model's ability to understand and model concept prerequisite relations. Extensive experiments on three publicly benchmark datasets demonstrate that DGCPL achieves state-of-theart performance for learning concept prerequisite relations.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62377009, 62407013 and 62207011.