Fine-Tuning Graph Neural Networks via Graph Topology Induced Optimal Transport
Fine-Tuning Graph Neural Networks via Graph Topology Induced Optimal Transport
Jiying Zhang, Xi Xiao, Long-Kai Huang, Yu Rong, Yatao Bian
Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Main Track. Pages 3730-3736.
https://doi.org/10.24963/ijcai.2022/518
Recently, the pretrain-finetuning paradigm has attracted tons of attention in graph learning community due to its power of alleviating the lack of labels problem in many real-world applications. Current studies use existing techniques, such as weight constraint, representation constraint, which are derived from images or text data, to transfer the invariant knowledge from the pre-train stage to fine-tuning stage. However, these methods failed to preserve invariances from graph structure and Graph Neural Network (GNN) style models. In this paper, we present a novel optimal transport-based fine-tuning framework called GTOT-Tuning, namely, Graph Topology induced Optimal Transport fine-Tuning, for GNN style backbones. GTOT-Tuning is required to utilize the property of graph data to enhance the preservation of representation produced by fine-tuned networks. Toward this goal, we formulate graph local knowledge transfer as an Optimal Transport (OT) problem with a structural prior and construct the GTOT regularizer to constrain the fine-tuned model behaviors. By using the adjacency relationship amongst nodes, the GTOT regularizer achieves node-level optimal transport procedures and reduces redundant transport procedures, resulting in efficient knowledge transfer from the pre-trained models. We evaluate GTOT-Tuning on eight downstream tasks with various GNN backbones and demonstrate that it achieves state-of-the-art fine-tuning performance for GNNs.
Keywords:
Machine Learning: Sequence and Graph Learning
Data Mining: Mining Graphs
Machine Learning: Multi-task and Transfer Learning