CONC: Complex-noise-resistant Open-set Node Classification with Adaptive Noise Detection
Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 5481-5489.
https://doi.org/10.24963/ijcai.2024/606
Keywords
- Data Mining: DM: Anomaly/outlier detection
- Data Mining: DM: Applications
- Machine Learning: ML: Classification
Abstract
As a popular task in graph learning, node classification seeks to assign labels to nodes, taking into account both their features and connections. However, an important challenge for its application in real-world scenarios is the presence of newly-emerged out-of-distribution samples and noisy samples, which affect the quality and robustness of learned classifiers. Out-of-distribution (OOD) samples are often found in both the training and testing phases. Such samples don’t belong to any known categories. These OOD samples are considered as outliers (OOD noise) when they appear during training, and are recognized as open-set samples during the testing. Meanwhile, in-distribution (IND) noisy data, i.e., known class samples with wrong labels, are also prevalent and inevitably degrade a model’s performance. The challenge of open-set learning with complex IND and OOD noise remains largely unexplored, particularly when dealing with non-IID graph data. To address these challenges, this paper introduces a novel complex-noise-resistant open-set node classification approach, designed for open-set graph data containing both IND and OOD noisy nodes. Specifically, a trustworthiness learner is adopted to learn the trustworthiness rates of the feature and label for each node while a decoder and an open-set classifier are trained to reconstruct the structure of a node and to predict its category simultaneously with the guidance of node trustworthiness. The experimental results demonstrate the superiority of our method.
- [Bendale and Boult, 2016] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In IEEE con- ference on computer vision and pattern recognition, pages 1563-1572, 2016.
- [Chereda et al., 2019] Hryhorii Chereda, Annalen Bleck- mann, Frank Kramer, Andreas Leha, and Tim Beissbarth. Utilizing molecular network information via graph con- volutional neural networks to predict metastatic event in breast cancer. In GMDS, pages 181-186, 2019.
- [Denouden et al., 2018] Taylor Denouden, Rick Salay, Krzysztof Czarnecki, Vahdat Abdelzad, Buu Phan, and Sachin Vernekar. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02765, 2018.
- [Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In Interna- tional conference on machine learning, pages 1263-1272.
- PMLR, 2017.
- [Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Neural information processing systems, 30, 2017.
- [Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
- [Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural informa- tion processing systems, 33:22118-22133, 2020.
- [Huang et al., 2022] Hongzhi Huang, Yu Wang, Qinghua Hu, and Ming-Ming Cheng. Class-specific semantic re- construction for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 4214 -4228, 2022.
- [Jiang et al., 2018] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. In International conference on machine learning, pages 2304-2313. PMLR, 2018.
- [Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
- [Lee et al., 2019] Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In In- ternational conference on machine learning, pages 3763- 3772. PMLR, 2019.
- [Liang et al., 2017] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690, 2017.
- [Liu et al., 2021] Zemin Liu, Yuan Fang, Chenghao Liu, and Steven CH Hoi. Relative and absolute location embedding for few-shot node classification on graph. In AAAI Con- ference on Artificial Intelligence, volume 35, pages 4267- 4275, 2021.
- [Malach and Shalev-Shwartz, 2017] Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Advances in neural information processing systems, 30, 2017.
- [Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learn- ing with noisy labels. Advances in neural information pro- cessing systems, 26, 2013.
- [Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 1944-1952, 2017.
- [Reed et al., 2014] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy la- bels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
- [Ren et al., 2018] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on ma- chine learning, pages 4334-4343. PMLR, 2018.
- [Song et al., 2019] Hwanjun Song, Minseok Kim, and Jae- Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pages 5907-5915. PMLR, 2019.
- [Sun et al., 2020a] Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13480-13489, 2020.
- [Sun et al., 2020b] Zeren Sun, Xian-Sheng Hua, Yazhou Yao, Xiu-Shen Wei, Guosheng Hu, and Jian Zhang. Crssc: salvage reusable samples from noisy data for robust learn- ing. In Proceedings of the 28th ACM International Con- ference on Multimedia, pages 92-101, 2020.
- [Sun et al., 2022] Zeren Sun, Fumin Shen, Dan Huang, Qiong Wang, Xiangbo Shu, Yazhou Yao, and Jinhui Tang. Pnp: Robust learning from noisy labels by probabilistic noise prediction. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 5311-5320, 2022.
- [Tang et al., 2022] Mingyue Tang, Carl Yang, and Pan Li. Graph auto-encoder via neighborhood wasserstein recon- struction. arXiv preprint arXiv:2202.09025, 2022.
- [Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
- [Wang et al., 2018] Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 8688-8696, 2018.
- [Wei et al., 2020] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13726-13735, 2020.
- [Wong et al., 2021] Randy Yee Man Wong, Christy MK Cheung, Bo Xiao, and Jason Bennett Thatcher. Standing up or standing by: Understanding bystanders' proactive re- porting responses to social media harassment. Information Systems Research, 32(2):561-581, 2021.
- [Wu et al., 2020] Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: Open-world graph learning. In IEEE interna- tional conference on data mining, pages 681-690, 2020.
- [Wu et al., 2021] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Ngc: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 62-71, 2021.
- [Wu et al., 2024] Man Wu, Xin Zheng, Qin Zhang, Xiao Shen, Xiong Luo, Xingquan Zhu, and Shirui Pan. Graph learning under distribution shifts: A comprehensive sur- vey on domain adaptation, out-of-distribution, and contin- ual learning. arXiv preprint arXiv:2402.16374, 2024.
- [Yang et al., 2022] Yijun Yang, Ruiyuan Gao, and Qiang Xu. Out-of-distribution detection with semantic mismatch un- der masking. In Computer Vision-ECCV 2022: 17th Eu- ropean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 373-390. Springer, 2022.
- [Yao et al., 2021] Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo- src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5192-5201, 2021.
- [Yu et al., 2017] Yang Yu, Wei-Yang Qu, Nan Li, and Zimin Guo. Open-category classification by adversarial sample generation. In International Joint Conference on Artificial Intelligence, pages 3357-3363, 2017. IJCAI
- [Zhang et al., 2018] Qinzhe Zhang, Jia Wu, Qin Zhang, Peng Zhang, Guodong Long, and Chengqi Zhang. Dual influence embedded social recommendation. World Wide Web, 21:849-874, 2018.
- [Zhang et al., 2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
- [Zhang et al., 2022a] Qin Zhang, Shangsi Chen, Dongkuan Xu, Qingqing Cao, Xiaojun Chen, Trevor Cohn, and Meng Fang. A survey for efficient open domain question answer- ing. arXiv preprint arXiv:2211.07886, 2022.
- [Zhang et al., 2022b] Qin Zhang, Qincai Li, Xiaojun Chen, Peng Zhang, Shirui Pan, Philippe Fournier-Viger, and Joshua Zhexue Huang. A dynamic variational framework for open-world node classification in structured sequences. In 2022 IEEE International Conference on Data Mining (ICDM), pages 703-712. IEEE, 2022.
- [Zhang et al., 2023] Qin Zhang, Zelin Shi, Xiaolin Zhang, Xiaojun Chen, Philippe Fournier-Viger, and Shirui Pan. G 2 pxy: Generative open-set node classification on graphs with proxy unknowns. arXiv preprint arXiv:2308.05463, 2023.
- [Zhang et al., 2024] Qin Zhang, Hao Ge, Xiaojun Chen, and Meng Fang. Unsupervised multiple choices ques- tion answering via universal corpus. arXiv preprint arXiv:2402.17333, 2024.
- [Zheng et al., 2020] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi-attention network for traffic prediction. In AAAI Conference on Artificial Intelligence, volume 34, pages 1234-1241, 2020.
- [Zhu et al., 2022] Qi Zhu, Chao Zhang, Chanyoung Park, Carl Yang, and Jiawei Han. Shift-robust node classifi- cation via graph adversarial clustering. arXiv preprint arXiv:2203.15802, 2022.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
The task of node classification is to assign labels to unlabeled nodes in the graph. Numerous vital applications, including predicting traffic states [Zheng et al., 2020], identifying diseases[Chereda et al., 2019], and completing user profiles on social networks [Wong et al., 2021], are associated with this task. Using methods [Zhu et al., 2022; Liu et al., 2021;
∗Corresponding author.
Zhang et al., 2018] based on GNN has achieved state-of-theart performance in addressing this issue.
However, a common assumption in the design of these methods is the availability of the complete set of labels (known classes) during the training phase. Unfortunately, this assumption often fails in real-world situations due to unpredictable factors[Zhang et al., 2022a; Wu et al., 2024; Zhang et al., 2024]. For example, a GNN trained on existing data may be applied to data from scientific social networks, encountering researchers who contribute to new conferences that have not been observed in training. When encountering samples from unknown classes, conventional GNN methods [Hamilton et al., 2017; Kipf and Welling, 2016; Veliˇckovi´c et al., 2017] will incorrectly assign labels from known classes to these nodes, which can significantly affect
can accurately classify known classes and effectively reject unknown classes, a task known as the open set node classification problem [Zhang et al., 2022b].
Moreover, in practical scenarios, data annotation unavoidably introduces IND noise (samples of known classes with erroneous labels) and OOD noise (outliers), and fixing the corrupted labels and data is typically extremely costly and timeconsuming. The existence of IND and OOD noise can negatively affect GNNs. These models may overfit to the noise, even in scenarios where labels are randomly assigned [Zhang et al., 2021]. Many approaches have been introduced to boost the generalization capabilities of models trained with noisy labels. Many methods have been introduced to enhance generalization when learning with noisy labels [Han et al., 2018; Jiang et al., 2018; Natarajan et al., 2013; Malach and ShalevShwartz, 2017; Reed et al., 2014]. Some methods analyze the output predictions to identify mislabeled samples [Yao et al., 2021; Sun et al., 2020b; Wu et al., 2021] or reweight samples to reduce the impact of noisy labels [Patrini et al., 2017; Song et al., 2019; Ren et al., 2018]. However, these methods are specifically designed for handling IND noise. Several other approaches consider the presence of OOD noise within training data [Lee et al., 2019; Wang et al., 2018], assuming that clean samples are clustered whereas OOD samples are scattered across the feature space[Wu et al., 2021]. The practical challenge of simultaneously dealing with IND and OOD noise during training, along with encountering OOD instances in the testing stage, has been explored by a limited number of studies [Wu et al., 2021; Sun et al., 2022].
Therefore, a crucial issue for real-world applications is to develop a method for complex noise-resistant open-set node classification, which can achieve open-set recognition and alleviate the effects of noisy data. Until now most previous work has focused on IID data, but the problem becomes more complex for non-IID graph data. A graph contains different types of information, such as proximity and structure information, which makes the problem of robust open-set learning on noisy graph data especially challenging.
The contribution of this work is to propose a novel Complex-noise-resistant Open-set Node Classification method, named CONC, designed to learn a robust open-set classifier from noisy graph data. To achieve this, a trustworthiness learner is adopted to learn the trustworthiness rate of the feature and label of each node. Meanwhile, a graph autoencoder is conducted to reconstruct the structures of each node, including node features and node degrees by taking nodes' trustworthiness into consideration. The differences in structure reconstruction errors between noisy nodes and clean nodes in training and the one between known-class and unknown-class samples in testing [Denouden et al., 2018; Yang et al., 2022] are exploited as a valuable indicator for the identification of noisy data and openset samples [Sun et al., 2022; Huang et al., 2022; Sun et al., 2020a]. Furthermore, an open-set node classifier is trained with weighted cross entropy by considering the type of noise. Finally, CONC achieves good robustness to complex noisy graph data and good effectiveness for open-set node classification. As far as we are aware, CONC is the very early effort in robust open-set node classification in the presence of complex noise within graph data. Experiments on benchmark graph datasets demonstrate the superiority of CONC.
2 ProblemDefinition &Preliminary
This research investigates the open-set node classification problem within a graph. We represent the graph as G = (V,E,X), where V denotes a set of N nodes, specifically V = {vi|i = 1, . . . ,N}. E represents the set of edges connecting node pairs, defined as E = {ei,j|i, j = 1, . . . ,N. i ̸= j}. Each node's features are captured in the feature matrix X ∈ RN×s , with s representing the feature dimension. The graph's topology is encoded by the adjacency matrix A ∈ RN×N , where Ai,j = 1 indicates an edge between nodes vi and vj, and Ai,j = 0 otherwise. Node labels are represented by the matrix Y ∈ RN×C , where C signifies the number of known node classes. If node vi belongs to class c, then yi,c = 1, and yi,c = 0 otherwise.
In a typical closed-set node classification scenario, a GNN encoder, denoted as ϕb, processes node features X and the adjacency matrix A. This encoder aggregates neighborhood information to generate node representations. Subsequently, a classifier ϕc assigns these representations to C predefined classes. The GNN encoder and the classifier are optimized to minimize the expected risk [Yu et al., 2017]:f ∗ = argminf∈HE(x,y)∼Dte I(y ̸= f(ϕb, ϕc; x,A)). This formulation assumes that the testing data Dte and the training data Dtr share the same feature and label spaces. Here, H represents the hypothesis space, and I(·) denotes the indicator function. This function can generally be optimized using cross-entropy to distinguish known categories.


In the open-set node classification problem, given a graph G = (V,E,X), Dtr = (X, Y ) represents the training node data. The test nodes are denoted by Dte = (Xte, Yte), where Xte = S ∪ U and Yte = {1, . . . , C, C + 1, . . .}. The set S is the nodes that belong to seen classes that already appeared in Dtr and U is the set of nodes that do not belong to any seen class (i.e., unknown class nodes). The objective of open-set node categorization is to develop a (C+1)-class classifier ¯ ϕc such that f(ϕb, ϕc;Xte,A) : {Xte,A} → Y, by minimizing the expected risk [Yu et al., 2017]:
where Y = {1, . . . , C, unknown}.
In the problem of open-set node classification with complex noise (IND and OOD noise), given a graph G = (V,A,X), Dtr = {(X, Y ), Y = S} = {xi, yi}Ni=1 is the training set and yi is the assigned label for node xi. Dte = {(Xte, Yte), Yte = S ∪ U} is the test set. Within the training set Dtr, each instance-label pair (xi, yi), where 1 ≤ i ≤ N, falls into one of three distinct categories based on the correspondence between its assigned label and its ground truth label, denoted as y∗ i : clean sample: the assigned label and the ground truth label are identical, meaning yi = y∗ i . IND noise sample: the assigned label deviates from the ground truth, but the true label still resides within the set of known classes (S). This can be expressed as yi ̸= y∗ i , y∗ i ∈ S. OOD noise sample: the assigned label not only differs from the ground truth but also does not correspond to any known class label. Formally, yi ̸= y∗ i , y∗ i /∈ S. Furthermore, the strict setting of open-set classification imposes an additional constraint: the ground truth label of OOD noise samples must not belong to the set of unseen classes (U) in the test set, ensuring no overlap. This is represented as y∗ i /∈ U. The ultimate objective is to train a noise-robust open-set node classifier capable of minimizing the expected risk as formulated in Eq. (1).
3 Methodology
To address the issue of open-set node classification with complex IND and OOD noise, we introduce a novel framework, named Complex noise-resistant Open-set Node Classification (CONC), to learn an effective open-set node classifier from noisy graph data. The framework, as shown in Fig.1, first adopts a k-layer GNN-based encoder ϕb, which takes an adjacency matrix A and node features X as input, outputs node representations Z = {zi}Ni=1 , followed by a trustworthiness learner ψt and a decoder ψd. The trustworthiness learner ψt is to learn two rates of a sample (xi, yi) with respect to its feature and label, to help with the IND and OOD noise detection, respectively. The decoder ψd is to learn the reconstruction of a node and its structure with the guidance of feature trustworthiness, to constrain the learning of node embeddings. Meanwhile, an open-set node classifier ψc to predict the class labels of nodes is learned with specific designs towards different trustworthiness rates and noise types. These modules are optimized under the constraints of consistency loss, reconstruction loss and classification loss, simultaneously.
3.1 Sample Trustworthiness Learning
A prevalent strategy to address noise involves identifying clean samples through a selection process that has been designed in advance [Sun et al., 2022], which has achieved promising results by identifying low-loss samples as clean data, building upon the observation that DNNs are inclined to learn simple patterns before overfitting to noisy data. However, these methods often make hard decisions on a sample, either as a clean sample with 100% trust or as a noise sample by directly discarding it. For example, NGC [Wu et al., 2021], CRSSC [Sun et al., 2020b] and Jo-SRC [Yao et al., 2021] utilize a threshold to discard the selected noisy samples, and the remained samples are supposed to be completely clean data. Similarly, Co-teaching [Han et al., 2018] and JoCoR [Wei et al., 2020] estimate a noise ratio to make noise removal and clean data selection. This arbitrary approach will result in a waste of resources since a certain amount of clean sample would be inevitably discarded, and also an amount of remained "clean" data would still contain more or less noise and should not be learned thoroughly. Therefore, in this paper, we try to achieve adaptive noise detection by learning two trustworthiness rates for each sample in terms of its features and its label respectively, and utilize the trustworthiness rates to guide the reconstruction learning and open-set classifier learning.
Specifically, to address the problem of sample trustworthiness learning, we adopt a probabilistic end-to-end learning approach. We propose to train a network, called trustworthiness learner ψt, to learn the rates ri = (rx i , ry i ) that represent the trustworthiness of each node vi with respect to its features and its label:
where σ(·) denotes sigmoid function. rx i = ri,0 represents the trustworthiness of node vi's features to reduce the risk of it being OOD noise, and ry i = ri,1 represents the trustworthiness of node vi's label to reduce the risk of it being IND noise. However, it's difficult to optimize the trustworthiness learner as the absence of ground-truth supervision to guide the process. Thus, we first adopt a label consistency loss to restrict the learning of the label trustworthiness rate, i.e.,
where Ni is the 1-hop neighbourhood of node vi. We will further optimize the learner in an end-to-end manner, integrating the optimization processes of both reconstruction learning and open-set classifier learning.
3.2 Adaptive Reconstruction Learning
To improve the recognition of open-set nodes and reduce the impact of noisy data, we learn the reconstruction of the node and its neighborhood size by taking the node's trustworthiness into consideration. By doing so, the reconstruction error can be leveraged to assist open-set recognition and noise reduction. Specifically, following the k-layer GNN encoder ϕb, we build a decoder ψd to simultaneously reconstruct the node features and node degrees, where the combination of these two terms can represent a larger receptive field of a node. This can be advantageous for open-set recognition, as known classes have been extensively studied and have lower reconstruction errors than unknown classes. Meanwhile, by considering the trustworthiness of each sample, the model would learn more about the clean data and adaptively avoid overfitting the noise by learning less about the noisy data.
Specifically, given a node vi ∈ V , the k-hop neighborhood of node vi is defined as the set of nodes reachable from vi through shortest paths with a maximum length of k. We denote the 1-hop neighborhood of node vi as Ni, and its degree as di. The encoder-decoder consists of encoder ϕb and decoder ψd. The encoder ϕb can be any message-passing GNNs [Gilmer et al., 2017], and learns node embeddings. The decoder ψd take node embeddings {zi}Ni=1 as input, which further consists of two parts: ψf and ψg. ψf decodes the node features, while ψg decodes the degrees.
For every node vi ∈ V , after being encoded by a GNN layer in ϕb, using first-hop neighborhood reconstruction as an example, its node representation h(1) i receives information from h(0) i = xi and its neighbor representations H(0) Ni =
{h(0) j = xj|vj ∈ Ni}. We aim to minimize the reconstruction loss for both vi and its neighborhood, that is, L(1) rec,i = L(1) i ((h(0) i ,H(0) Ni , ψd(h(1) i )), where h(1) i = ϕb(h(0) i ,H(0) Nvi ).
In this expression, L(1) rec,i denotes the overall reconstruction loss with the 1-hop neighborhood and can be split into two components: self-feature reconstruction loss and neighborhood size reconstruction loss (the degree of node vi) , which can be expressed as:
The two losses push the model to learn the characteristics of node features and the main information of node structure in depth by taking full advantage of neighborhood features. Further, we take feature trustworthiness into consideration, push the model to learn more about the clean features and reduce the impact of noise during the model learning. Thus, the overall reconstruction loss of vi by taking advantage of its k-hop neighborhood and its trustworthiness can be expressed as follows:
where zi = h(k) i is the embedding of xi learned by the klayers encoder ϕb. λs is a non-negative hyperparameter used to balance the weights of the two losses. rx i is the feature trustworthiness rate of node vi. And the total reconstruction loss Lrec of graph G = (X, A, Y ) is
The minimization of the total reconstruction loss Lrec ensures encoder ϕb learning informative node embeddings zi, i = 1, . . . ,N, and push the decoder ψd reconstructing node features and degrees, and also make the sample trustworthiness learner ψt learn accurate trustworthiness rates.
3.3 Robust Open-set Classifier Learning
To deal with unexpected unknown samples that appear during testing, we train another parallel network ψc, an open-set node classifier, to predict the category label of each node:
where C is the number of known classes.
To mitigate the influence of noisy data, we use adjusted soft labels to supervise the training of the open-set classifier. We first identify the types of noisy samples based on the learned trustworthiness rates. Specifically, we define them as follows:
where τ1, τ2 are two hyper-parameters defined as thresholds. Then we adjust the label of node vi to ˜ yi according to different noise types, i.e.,
For clean samples, we directly inherit their original labels, while for IND noise we reduce the value of their non-zero elements since their assigned labels are not trustworthy. For OOD noise, we make the soft label as probability from union distribution with a parameter of ry i for the C known classes, since it does not belong to any known classes and is more likely to come from an unknown class.
Further, to effectively handle different types of noisy samples, we employ distinct loss functions for each type. For clean nodes, we utilize cross-entropy loss and incorporate an entropy regularization term:
where lce represents the cross-entropy loss and let denotes the entropy loss. pic refers to the predicted probability that xi is classified as belonging to class c, c = 1, . . . , C. ¯ pic = pic(¯xi) is the predicted probability of the reconstruction of xi, by feeding the embedding of its reconstructed feature ¯ zi into the open-set node classifier ψc. In the loss, the first term is to push the prediction being aligned with the adjusted label assignments, while the second term treats the outputs of original nodes and reconstructed nodes as predictions and targets to construct classification loss. The third term uses an entropy regularization term to make the prediction more certain.
For IND and OOD noisy samples, we conduct losses lind(xi, ˜ yi) and lood(xi, ˜ yi) similarly as follows:
where ϵ(z, T) = exp(z/T) ∑z′ exp(z′ /T) , which is inspired by ODIN [Liang et al., 2017], and T is a temperature scale parameter. For IND noisy nodes, where a well-trained model's predictions often surpass the reliability of given annotations, we apply a sharpening operation. This enforces greater confidence in predictions, thereby enhancing training. Conversely, OOD noisy nodes tend to confuse models due to their ground-truth categories falling outside the training distribution. By applying a flattening operation, we guide their predictions towards an approximately uniform distribution, ultimately boosting robustness and generalization performance[Sun et al., 2022]. In the experiment, we empirically set T = 0.1. Moreover, for OOD noise samples, we omit the entropy regularization term since there is little information on the unknown classes and it is hard to obtain high confidence for the sample belonging to open-set classes, thus we do not push the model to give unique highly confident decisions.
The classification loss Lcls is a combination of lclean, lind, and lood as a weighted loss based on the noise type predictions:
| Methods GCN soft GCN sig GCN soft τ | GCN sig τ | NWR | τ Openmax OpenWGL | G2Pxy | NGC | PNP | CONC | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5% F1 | 63.73 fail | 64.12 | 70.53 | 57.80 | 69.41 | 60.40 | 73.03 | 72.00 | 59.69 | 70.26 | 70.81 85.92 | ||
| AUROC | fail | 82.82 | 81.40 | 75.05 | 67.26 | 79.36 | 78.30 | 78.20 | 82.04 | ||||
| Cora F1 | 64.31 | 63.78 | 70.69 | 66.91 | 68.98 | 59.82 | 70.58 | 69.92 | 52.13 | 64.78 | 75.35 | ||
| 25% AUROC | fail | fail | 77.92 | 79.94 | 81.69 | 87.61 | 77.90 | 77.36 | 80.22 | 84.87 | 88.82 | ||
| 50% F1 | 60.96 | 59.73 | 62.53 | 62.93 | 61.93 | 53.50 | 63.59 | 57.52 | 62.70 | 66.29 | 76.14 | ||
| AUROC | fail | fail | 79.46 | 78.52 | 78.73 | 80.88 | 85.40 | 63.68 | 83.64 | 83.54 | 89.45 | ||
| 5% F1 | 39.03 | 37.74 | 59.13 | 59.93 | 52.28 | 33.23 | 59.87 | 52.40 | 59.44 | 59.99 | 63.52 | ||
| AUROC | fail | fail | 85.67 | 85.56 | 76.13 | 55.01 | 86.08 | 67.75 | 84.05 | 85.70 | 88.70 | ||
| Citeseer F1 | 38.30 | 38.02 | 23.04 | 57.79 | 48.51 | 36.70 | 57.42 | 53.53 | 55.42 | 59.83 | 60.27 | ||
| 25% AUROC | fail | fail | 84.50 | 82.26 | 72.59 | 75.68 | 82.34 | 53.97 | 80.46 | 84.03 | 86.31 | ||
| 50% F1 | 30.39 | 31.71 | 35.41 | 35.54 | 35.65 | 24.83 | 40.46 | 41.92 | 56.39 | 51.81 | 55.75 | ||
| AUROC | fail | fail | 64.20 | 50.74 | 61.22 | 67.89 | 79.54 | 38.03 | 77.43 | 80.40 | 83.38 | ||
| F1 | 75.44 | 76.50 | 76.63 | 71.70 | 75.43 | 62.68 | 74.09 | 83.45 | 77.61 | 83.56 | 82.77 92.57 | ||
| 5% AUROC | fail | fail | 83.14 | 86.61 | 89.57 | 81.65 | 87.39 | 82.95 | 85.57 | 85.55 | |||
| Coauthor-CS F1 | 74.58 | 75.19 | 81.92 | 73.38 | 65.64 | 66.38 | 72.98 | 67.21 | 69.18 | 79.37 | 81.03 | ||
| 25% AUROC | fail | fail | 87.77 | 88.35 | 88.69 | 80.62 | 86.35 | 86.03 | 85.11 | 85.47 | 91.31 | ||
| 50% F1 | 72.93 | 60.77 | 68.87 | 52.63 | 78.66 | 59.97 | 54.70 | 72.96 | 63.06 | 68.40 | 79.14 | ||
| AUROC | fail | fail | 75.02 | 83.97 | 87.58 | 81.48 | 85.13 | 72.87 | 84.27 | 84.85 | 92.16 | ||
| F1 | 27.39 | 26.64 | 25.43 | 25.91 | 26.90 | 16.47 | 27.77 | 25.46 | 27.36 | 28.48 | 33.09 | ||
| 5% AUROC | fail | fail | 73.59 | 72.12 | 76.73 | 60.17 | 76.35 | 72.09 | 75.91 | 76.96 | 77.83 | ||
| Ogbn-arxiv 25% F1 | 26.84 | 25.51 | 25.84 | 23.29 | 26.44 | 15.60 | 26.23 | 22.08 | 26.24 | 27.67 | 31.00 | ||
| AUROC | fail | fail | 72.89 | 73.05 | 75.91 | 54.93 | 73.50 | 66.24 | 74.49 | 75.48 | 76.61 | ||
| 50% F1 | 25.74 | 24.28 | 23.84 | 21.29 | 22.64 | 12.42 | 22.90 | 15.95 | 24.03 | 24.94 | 28.98 | ||
| AUROC | fail | fail | 71.21 | 71.86 | 74.16 | 52.94 | 72.01 | 57.98 | 72.87 | 73.28 | 75.30 | ||
lclean, lind and lood are shown in Eq. (10), (11) and (12), respectively.
Based on the definitions and designs presented above, we train CONC in an end-to-end manner by optimizing the following overall objective loss function:
where γ1, γ2 are hyperparameters that adjust the relative importance of different loss terms. lcons and lrec are shown in Eq. (3) and (6), respectively. To fully attain the goal of robust open-set node classification, the framework also needs to handle the presence of unknown class samples in testing. To perform hard predictions, we utilize a probability threshold τ . If maxc ˆ pi,c < τ , then the test point xi is considered a sample of an unknown class.
4 Experiments
We designed experiments to evaluate CONC, focusing primarily on: comparison of open-set classification, robustness analysis, and ablation study. The relevant code will be made publicly accessible online.
4.1 Experimental Setup
Dataset and metrics. To evaluate the performance of the proposed framework for robust open-set node classification, we performed experiments on four widely-used citation network benchmark datasets: Cora, Citeseer, Coauthor-CS, and Ogbn-arxiv. The datasets' statistics are presented in the Appendix. Aligning with Wu et al. [Wu et al., 2021] study, we utilize macro-F1 and AUROC metrics to quantify performance.
Implementation Details. For CONC, we use GCN [Kipf and Welling, 2016] as the backbone network for the encoder ϕb, with four hidden layers of 128 dimensions. We use a fourlayer GCN layer plus a linear layer to implement the trustworthiness learner ψt. The feature decoder ψf and the degree decoder ψg are configured with three-layer and four-layer multilayer perceptron with a dimension of 128. To train the openset node classifier ψc, an additional linear layer is added for it. We employed a grid search to determine the optimal values for thresholds τ, τ1, τ2, exploring the range from 0 to 1 with a step size of 10-1 . The CONC framework is implemented using PyTorch, and network optimization leverages adaptive moment estimation with a learning rate of 10-3 . Balance parameters γ1 and γ2 were also identified through a grid search, considering values from 10-3 to 101 with a step size of 101 .
To maintain consistency across experiments, both the baseline and proposed methods utilized the same training, validation, and testing datasets.
Test settings. To assess open-set node classification performance, each dataset was split into known and unknown classes, holding out several classes as unknowns for testing. Among the known classes, 70% of nodes were allocated for training, 10% for validation, and the remaining 20% for testing.
| 5% far OOD noise | 25% far OOD noise | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Methods | Cora | Citeseer | Coauthor-CS | Ogbn-arxiv | Cora | Citeseer | Coauthor-CS | Ogbn-arxiv | ||||||||
| F1 | AUROC | F1 | AUROC | F1 | AUROC | F1 | AUROC | F1 | AUROC | F1 | AUROC | F1 | AUROC | F1 | AUROC | |
| GCN soft | 64.67 | fail | 39.04 | fail | 60.36 | fail | 29.74 | fail | 63.12 | fail | 39.04 | fail | 77.08 | fail | 28.91 | fail |
| GCN sig | 64.11 | fail | 41.33 | fail | 69.32 | fail | 29.33 | fail | 63.44 | fail | 40.78 | fail | 65.94 | fail | 27.72 | fail |
| GCN soft τ | 71.81 | 74.61 | 58.57 | 82.35 | 65.81 | 78.37 | 27.72 | 66.81 | 70.38 | 75.13 | 56.09 | 81.61 | 60.48 | 72.37 | 25.91 | 64.69 |
| GCN sig τ | 62.65 | 77.54 | 54.60 | 84.58 | 54.23 | 86.73 | 27.04 | 65.97 | 69.23 | 76.38 | 57.41 | 81.81 | 57.75 | 84.31 | 24.97 | 65.04 |
| Openmax | 28.37 | 79.06 | 26.64 | 81.17 | 66.90 | 82.10 | 20.15 | 68.91 | 31.41 | 84.40 | 15.55 | 79.03 | 67.23 | 85.07 | 18.65 | 53.31 |
| OpenWGL | 71.83 | 76.76 | 58.51 | 84.78 | 76.38 | 87.24 | 29.18 | 68.52 | 67.87 | 74.50 | 57.28 | 82.75 | 75.71 | 85.10 | 26.46 | 66.87 |
| G 2Pxy | 69.24 | 80.67 | 43.26 | 62.53 | 66.98 | 69.57 | 27.71 | 64.07 | 66.54 | 75.26 | 41.67 | 60.95 | 70.45 | 68.34 | 27.03 | 64.05 |
| NWR τ | 71.94 | 76.58 | 48.35 | 75.06 | 79.46 | 87.65 | 24.61 | 71.07 | 68.79 | 80.67 | 48.08 | 71.67 | 54.77 | 83.76 | 24.12 | 67.43 |
| NGC | 57.58 | 78.57 | 52.30 | 78.07 | 66.96 | 84.47 | 24.36 | 69.73 | 59.86 | 76.88 | 55.41 | 81.22 | 64.84 | 83.96 | 27.73 | 65.86 |
| PNP | 72.21 | 79.51 | 60.22 | 83.93 | 77.50 | 87.42 | 30.17 | 68.38 | 72.19 | 77.47 | 57.58 | 81.94 | 81.34 | 84.23 | 28.40 | 66.38 |
| CONC | 73.64 | 77.57 | 60.78 | 85.13 | 84.07 | 87.73 | 34.46 | 69.45 | 72.71 | 77.60 | 59.77 | 83.29 | 82.29 | 85.46 | 35.14 | 68.25 |
To evaluate the CONC framework's performance on graph data with various types of noise, we tested it with mixed IND and OOD noise. Within the training set, we randomly selected 5%, 25%, or 50% of known class samples as IND noise, replacing their ground truth labels with incorrect labels from known classes. For OOD noise, we used nodes absent from both the training set's known classes and the test set's unknown classes, then randomly assigned them known class labels. The final training set is formed by combining clean data (correctly labeled known class samples), IND noise (known class samples with incorrect labels), and OOD noise (distant unknown class samples with incorrect known class labels). The test set consists of samples from both known and unknown classes, with the unknown classes being distinct from those in the OOD noise. All experiments were conducted using an inductive learning approach, meaning that withholding any information about the test set's unknown classes (e.g., feature xi or side information) during both training and evaluation.
Baselines. To validate the effectiveness of CONC, three categories of baselines are included in our experiments:
- 1) Closed-set classification methods: GCN soft and GCN sig. These methods utilize GCN [Kipf and Welling, 2016] with different output layers: GCN soft employs a softmax layer, while GCN sig uses multiple 1-vs-rest sigmoid layers. They lack the ability to recognize unknown classes.
- 2) Open-set classification methods: GCN soft τ, GCN sig τ, NWR τ [Tang et al., 2022], Openmax [Bendale and Boult, 2016], OpenWGL [Wu et al., 2020] and G 2Pxy [Zhang et al., 2023]. Specifically, GCN soft τ and GCN sig τ are based on GCN soft and GCN sig, NWR τ is a novel graph representation learning method based on neighborhood reconstruction, all of them use a probability threshold chosen from {0.1, 0.2, . . . , 0.9} to achieve open-set recognition. Openmax is an open-set recognition model based on "activation vectors". OpenWGL and G 2Pxy are two methods for open-set node classification on graph data, they lack robust learning capabilities.
- 3) Robust open-set classification methods: NGC [Wu et al., 2021] and PNP [Sun et al., 2022]. NGC is a robust openset classification method for image data that uses predictive confidence and geometric structure information to select clean samples for training. PNP is a closed-set robust classification method for image data, which uses data augmentation to help identify noisy samples. we add a threshold to facilitate open-set recognition.
Note that before feeding graph data into models that are not equipped to process such data, the data are first embedded by a GCN. A detailed introduction is provided in the Appendix.
4.2 Open-set Node Classification with Complex Noisy Graph Data
Considering the complexity of real-world scenarios and noise data varies between tasks, we evaluated our proposed openset classification model under various scenarios: IND noise and two types of OOD data - near OOD and far OOD. Here, OOD data includes the training set's OOD noise and the test set's out-of-distribution samples from unknown classes.
Open-set classification with IND noise and near OOD data. Following the OpenWGL's setting [Wu et al., 2020], each dataset's final class was held out as the unknown class for testing. The penultimate class, served as OOD noise injected into the training set after being randomly reassigned known class labels. The remaining classes were designated as known classes. Within the training set, known class samples were re-labeled with incorrect known class labels at rates of 5%, 25%, and 50%, respectively, constituting IND noise.
Table 1 presents the macro-F1 and AUROC scores for open-set node classification with near OOD noise and varying proportions of IND noise. The results show that CONC typically achieves superior performance in benchmark evaluations, indicating its effectiveness in distinguishing between known and unknown classes even amidst substantial noise during training. Compared to the second-best method (PNP), CONC showed an average improvement of 4.37% in F1 score and 3.85% in AUROC on three datasets.
Moreover, we analyzed the detailed classification accuracy regarding known and unknown classes. We found that, compared to closed-set classifiers, in order to gain unknown class detection capability, the performance of classifying known classes slightly decreased, dropping from an average of 80.67% (GCN soft) to 68.54% (CONC), while the accuracy of detecting unknown classes significantly increased from 0% to an average of 77.56%. Additionally, compared to other methods for open-set node classification like G 2Pxy, CONC has achieved an average increase of 8.15%, 15.00% and 8.78% in known class accuracy, unknown class accuracy and overall accuracy, respectively. The F1 score has improved by 8.62% on average. Details can be found in the Appendix.

Open-set node classification with IND noise and far OOD data. To explore the impact of far OOD noise, we selected OOD samples from different datasets as the noise source. More specifically, we randomly selected instances from the first two classes of Pubmed [Hu et al., 2020] and integrated them with data from Cora, Citeseer, Coauthor-CS, and Ogbnarxiv. This process created training sets containing far OOD noise at rates of 5% and 25%. For each node from Pubmed, a known class label was randomly assigned, and edges were added connecting the node to its k-nearest neighbors, where k is a random integer between 1 and 5. To evaluate CONC's performance on far open-set samples, we incorporated a proportional selection of samples from the remaining Pubmed categories into the test set. The remaining settings remain the same as the near OOD data experiment, except for the IND noise rate, which was set to 5%.
Table 2 presents the results of open-set node classification under far OOD noise conditions. The results indicate that CONC generally outperforms the baselines. Detailed classification accuracy for both known and unknown classes in the presence of far OOD noise can be found in the Appendix.
4.3 Robustness Analysis
In this part, we evaluated the robustness of CONC for the open-set node classification task under various levels of IND and OOD noise. We first tested CONC's reaction to different IND noise rates while maintaining a constant OOD noise rate, following the experimental setup of Table 1. IND noise rates were varied across 0%, 5%, 25%, 50% and 75%. Results on the Citeseer and Coauthor-CS datasets are depicted in Fig.2(a) and Fig.2(b), respectively. We observe that CONC exhibits relatively stable performance when the IND noise rate remains below a certain threshold, such as 50%. This demonstrates CONC's robustness against IND noise. However, exceeding this threshold leads to a sharp decline in CONC's performance.
| Methods | Cora | Citeseer | Coauthor-CS | |||
|---|---|---|---|---|---|---|
| F1 | AUROC | F1 | AUROC | F1 | AUROC | |
| CONC¬rec | 69.19 | 79.83 | 58.11 | 82.72 | 78.78 | 85.37 |
| CONC¬cons | 68.75 | 79.45 | 59.19 | 83.40 | 78.37 | 84.11 |
| CONC sharp | 70.21 | 81.43 | 60.27 | 85.09 | 80.04 | 85.62 |
| CONC | 70.81 | 85.92 | 63.52 | 88.70 | 82.77 | 92.57 |
Furthermore, we evaluated CONC's performance under different OOD noise rates while maintaining a constant IND noise rate of 5%. Far OOD noise rates were varied across 0%, 5%, 25%, 50% and 75%. Results for the Citeseer and Coauthor-CS datasets are presented in Fig.2(c) and Fig.2(d), respectively. The results show that CONC maintains stable performance despite substantial OOD noise in training data, indicating its strong robustness.
4.4 Ablation Study
We also compared different variants of CONC within the ablation study to evaluate the impact of its primary modules.
- CONC¬rec: a variant with adaptive reconstruction learning module is removed.
- CONC¬cons: a variant with label consistency loss is removed.
- CONC sharp: in this variant, we donate the predicted noise type of node vi as pi = [pclean i , pind i , pood i ] ∈ {0, 1}3 which is a one-hot vector. Then we only keep one item for each node in classification loss lcls, i.e., Lcls = 1N ∑Ni=1 (pclean i lclean,i + pind i lind,i +pood i lood,i).
Table 3 lists the performance of the proposed method and its three variants. The results show both reconstruction and trustworthiness learning modules are crucial. The learned trustworthiness rates provide valuable insights for improving open-set node classification.
5 Conclusions
This article introduces a novel method for robust open-set node classification on graphs with complex IND and OOD noise. The method uses a graph auto-encoder for structure reconstruction and two networks for predicting node category and trustworthiness. These modules are jointly optimized. CONC, to the best of our knowledge, represents an early effort in this area and demonstrates promising results compared to existing approaches.
Acknowledgments
This research was supported by the National Natural Science Foundation of China (62206179, 62273241), the Natural Science Foundation of Guangdong Provincial (2022A1515010129, 2024A1515011946), and the University Stability Support program of Shenzhen (20220811121315001).
@inproceedings{ijcai2024p606,
title = {CONC: Complex-noise-resistant Open-set Node Classification with Adaptive Noise Detection},
author = {Zhang, Qin and Lu, Jiexin and Li, Xiaowei and Wu, Huisi and Pan, Shirui and Chen, Junyang},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence, {IJCAI-24}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Kate Larson},
pages = {5481--5489},
year = {2024},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2024/606},
url = {https://doi.org/10.24963/ijcai.2024/606},
}
TY - CPAPER TI - CONC: Complex-noise-resistant Open-set Node Classification with Adaptive Noise Detection AU - Zhang, Qin AU - Lu, Jiexin AU - Li, Xiaowei AU - Wu, Huisi AU - Pan, Shirui AU - Chen, Junyang T2 - Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2024 SP - 5481 EP - 5489 DO - 10.24963/ijcai.2024/606 UR - https://doi.org/10.24963/ijcai.2024/606 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: