Task-Guided and Semantic-Aware Ranking for Academic Author-Paper Correlation Inference

1 Introduction

Due to the growing evolution of scientific research and increasing data collections by various online services such as Google Scholar, Microsoft Academic or AMiner, the problems of mining big scholarly data have gained a lot of attention in the past decade. Examples include scientific impact modeling and prediction [Wang et al., 2013; Shen et al., 2014; Dong et al., 2015], heterogeneous bibliographic network analysis [Sun et al., 2012; Huang et al., 2016; Chen and Sun, 2017], etc.

In this work, we consider the problem of author-paper correlation inference in big scholarly data. Specifically, given an author's previous correlated papers (e.g., publications or references), we would like to effectively infer the potential relevant works for him/her, such that the author will interact (e.g., cite) with those papers in the future. Solutions of the problem bring important implications to researchers with different knowledgeable levels. For example, an effective algorithm provides suitable academic paper reading lists for new PhD students with little historical feedback in database, or helps active scientists track the related or following works of their previous publications. In addition, it can be a good reference for recommender system design in digital libraries such as Elsevier or Springer.

As one of the representative solutions, supervised leaning algorithms can be applied to predict the correlation score between author and paper, as they were used in 2013 KDD cup author-paper pair identification challenge [Efimov et al., 2013; Li et al., 2015]. However, such methods heavily rely on time and memory consuming feature engineering, and the extracted features may be too simple to capture complicated relations. In recent years, unlike traditional supervised leaning, several network embedding models [Perozzi et al., 2014; Grover and Leskovec, 2016; Chen and Sun, 2017; Dong et al., 2017] have been proposed to automatically learn nodes' representations that can be further used for various applications in scholarly data such as author-paper correlation inference. Although the proximity among nodes is preserved by dense vectors, most of the existing embedding models suffer from following two drawbacks:

• They produce general purpose embeddings that are independent of task, no matter what kind (homogeneous or heterogeneous) of networked data is used. However, when it comes to the author-paper correlation inference problem, nodes should be embedded under the guidance of target for generating task-specific representations.
• Even though they take account of the task for embedding generation, e.g., TaskE [Chen and Sun, 2017], they are purely based on network structure and out of content awareness. Actually, content (e.g., abstract of paper) contains useful semantic information that should be used for encoding nodes into a better feature space.

To address the above issues and solve the given problem, we propose a task-guided and semantic-aware ranking model. First, we model the historical interactions among correlated author-paper pairs via pairwise ranking according to the specific task. Next, we introduce the gated recurrent neural network to encode paper's content, and combine the obtained semantic embedding with author's latent feature to score each author-paper pair in ranking loss. Moreover, we design a heterogeneous relations integrative learning module to formulate the indirect correlations among authors and papers, and further augment the model. Finally, a relations sampling based mini-batch gradient descent algorithm is designed for model training. The main contributions of this paper are summarized as follows:

• We study the author-paper correlation inference problem in big scholarly data, which brings important implications to academic community.
• To solve the problem, we propose a model by jointly content semantic encoding and heterogeneous relations augmented ranking, and design the corresponding learning algorithm.
• We conduct extensive experiments to evaluate the performance of the proposed model on the well known AMiner dataset. The results show that our model significantly outperforms a number of baselines.

2 Problem

We introduce few notations that will be used throughout this paper. Specifically, we denote the sets of authors and papers as U and I, respectively. Let lu<T be the set of both author u's publications and references before a given timestamp T. Similarly, lu≥T represents u's papers and references after T. In this work, the correlated papers of each author are assumed as both publications and references in dataset. The problem is formalized as:

Academic Author-Paper Correlation Inference. Given lu<T of each author u ∈ U and the content (i.e., abstract) of each paper v ∈ I, the goal is to learn a model to rank all potential papers v′ ∈ I\lu<T for u, such that its top return papers are in lu≥T .

Note that each author can cite previous works or write new papers, thus the return papers can be published both before and after T. In addition, the overlapping between lu<T and lu≥T are removed from lu≥T , so that papers in lu≥T are never cited by u before T .

3 ProposedModel

In this section, we present how to design task-guided and semantic-aware ranking model for solving the problem, and use historical academic data to construct a heterogeneous network for capturing indirect author-paper relations which benefit and augment the model.

3.1 Pairwise Ranking with Gated Recurrent Neural Network

We model historical interactions among correlated authorpaper pairs via pairwise ranking optimization [Rendle et al., 2009]. Specifically, for a given author u, the correlated paper v ∈ lu<T should be ranked higher than the uncorrelated paper v′ ∈ I\lu<T . In other words, the relevance score su,v of 〈u, v〉 pair should be larger than that of 〈u, v′ 〉 pair as much as possible, leading to an author-paper pairwise ranking loss as follows:

where σ is the sigmoid function. For each author-paper pair 〈u, v〉, we introduce author u's latent feature qu ∈ Rd (d: the embedding dimension) and represent paper v as semantic embedding Epv ∈ Rd via content encoder f: Epv = f(pv), where pv represents paper v's content. The inner product of qu and f(pv) is used to measure the relevance score, i.e., su,v = qu T f(pv).

To encode papers' contents to fixed size embeddings Ep ∈ R|I|×d , we use the gated recurrent units (GRU), a specific type of recurrent neural network (RNN), which has been widely adopted for many applications such as machine translation [Cho et al., 2014]. Specifically, a paper is represented by a sequence of word embeddings: {x1, x2, · · · , xtmax }, where xt denotes the t-th word embedding pre-trained by word2vec [Mikolov et al., 2013] and tmax is the maximum length of paper's abstract. For each step t with the input of word embedding xt and hidden state vector ht-1, GRU computes the updated hidden state vector via ht = GRU(xt, ht-1), where GRU module is defined as:

where σ is the sigmoid function, the operator ◦ denotes element-wise multiplication, N and M are parameter matrices of GRU network, zt and rt are updated gate vector and reset gate vector, respectively. We apply the above GRU network to encode words' contextual embeddings h ∈ Rtmax×d and use a mean pooling layer to obtain the general semantic embedding of each paper. All of these steps construct the paper's content encoder f, as illustrated by Figure 1(a). Note that, we also explore other encoding architectures such as LSTM or attention-based GRU but obtain similar result. Thus we choose GRU since it has a concise structure for reducing training time.

Using the above GRU encoder for papers' embeddings and the authors' latent features, we further minimize the pairwise ranking loss (i.e., Eq. (1)) via gradient descent approach, as illustrated by Part-1 of Figure 1(b). The process leads to taskguided and semantic-aware ranking and we name it TSR.

3.2 Heterogeneous Relations based Integrative Learning Augmentation

TSR trains model by only using direct correlations, i.e., correlated author-paper pairs in lu<T for each author u. However, there are multiple indirect author-paper relations, which can be inferred from direct correlations and useful for improving TSR. Inspired by DeepWalk [Perozzi et al., 2014], we apply randomwalk to collect those indirect correlations. Figure 1(b) Part-2 gives the illustration. First, we use the correlated papers set lu<T (contains both author u's publications and references before T ) of each author u to create a heterogeneous Algorithm 1: Learning Framework of TSR+ input : Crank in training data and Chinge collected by random walk sampling on HetNet output: authors' latent features q, GRU encoder parameter matrices N and M (for papers' embeddings f(p)) 1 while not converged do 2 sample a mini-batch of (u, v, v′ ) triples in Crank; 3 sample a mini-batch of (u, v, v′ ) triples in Chinge; 4 accumulate the loss by Equation (6) ; 5 update the parameters by Adam Optimizer 6 end W be the set of all walks. The objective of augmented TSR (TSR+) is defined as the combination of TSR and HRIL:

where Lreg is the regularization term for avoiding overfitting. We denote all model parameters including GRU network coefficients (for generating papers' embeddings) and authors' latent features as Θ, and let Crank and Chinge be the sets of (u, v, v′ ) triples in Lrank and (u, v, v′ ) triples in Lhinge, respectively. Thereafter we can rewrite the objective of TSR+ as:

where parameter λ controls regularization penalty. To minimize the above objective function, we design a relations sampling based mini-batch Adam Optimizer [Kingma and Ba, 2014], as illustrated by the pseudocode in Algorithm 1.

4 Experiments

In this section, we conduct extensive experiments to evaluate the proposed model and various baselines. Case studies are also provided.

network (HetNet) with two kinds of nodes (author and paper) and two kinds of undirected edges (author write ----⇀ ↽---- write-1 pa-

per and paper cite ---⇀ ↽--- cite-1 paper). Then we perform random walk

sampling over HetNet to generate a set of node sequences. For example, in Figure (b) Part-2, a heterogeneous walk: wa ≡ {a → 1 → b → 2 → c → 4 → 3 → 5} is generated and the surrounding context of each node in wa implies different relations among authors and papers within wa. We use sub-sequence ˆ w1 ≡ {1 → b → 2 → c → 4 → 3 → 5} (window size equals to 3) centered at author c as an illustration. Besides the direct connections, e.g., author c writes paper 2, ˆ w1 also captures multiple indirect relations. For example, author c has indirect citation relation with paper 5 since s/he has citation relation with paper 3. Therefore, each heterogeneous walk contains both direct correlations and indirectly transitive relations among authors and papers.

To formulate indirect relations within each walk, we design a heterogeneous relations integrative learning module (HRIL) to augment TSR based on a reasonable assumption that the relevance scores of indirectly correlated author-paper pairs should be larger than those of uncorrelated pairs. Specifically, we introduce a hinge loss to formulate the difference of author u's correlations to indirectly correlated paper v and uncorrelated paper v′ :

where {x}+ = max(x, 0) and ξ is a positive margin value. A loss penalty will incur if the score of 〈u, v〉 is not at least ξ larger than that of 〈u, v′ 〉. Such formulation has been widely adopted in recent works [Chen and Sun, 2017; Zhang et al., 2017; 2018] for modeling preference differences. Thus the overall loss in each walk w is formulated as:

where τ is the window size of surrounding context used for relations extraction, and Iu indicates the position of u in w. Therefore constraining Lhinge obeys our assumption.

3.3 Model Training

We generate a plenty of walks rooted at each author node to collect indirect correlations among authors and papers, and let

4.1 Experimental Design

Dataset.

We use the public available dataset 1 AMiner [Tang et al., 2008] between 2005 and 2015, and remove the papers published in venues (e.g., workshop) with limited number of publications and the instances without content (i.e., abstract). Besides, considering most of researchers pay attention to papers published in top venues and each research area has its own community, we extract a subset data of six areas according to Google Scholar Metrics 2 , namely Artificial Intelligence (AI), Computer Vision (CV), Data Mining (DM), Databases (DB), Computational Linguistics (CL) and Information System (IS). For each area, we choose three top venues 3 that are considered to have influential publications. The main statistics of two datasets (AMiner-T and AMiner-F) are summarized in Table 1. Note that, there are some missing citations in the dataset. It makes the inference task more difficult due to potential noise in model training.

Baselines.

We compare TSR and TSR+ with eight baseline methods that span three categories: (1) feature-based supervised learning, (2) content-based ranking and (3) network embedding.

• Feature-based supervised learning. It first extracts author-paper paired features and then applies supervised learning algorithms to predict the relevance score of each author-paper pair. We extract 14 kinds of features (as shown in Table 2) and choose Bayesian Regressor (BayesR), Neural Network (NeuNet) and Random Forest (RandomF) as learning algorithms. For each correlated author-paper pair, we randomly sample 5 negative pairs to train model.
• Content-based ranking. It first encodes each paper's content (i.e., abstract) via language modeling and then applies pairwise ranking BPR [Rendle et al., 2009] to learn each author's latent feature. We employ two popular models word2vec [Mikolov et al., 2013] and paragraph vector (doc2vec) [Le and Mikolov, 2014] to learn papers' embeddings. Note that word2vec encodes embedding of each word in content, we connect the output of word2vec with a mean pooling layer to obtain general embedding of each paper.
• Network embedding. It learns embeddings of both authors and papers based on the structure of author-paper heterogeneous network (same as the HetNet in TSR+). We use both homogeneous model Deepwalk [Perozzi et al.,

1 https://aminer.org/citation

2 https://scholar.google.com/citations?view op=metrics intro&hl=en

3 AI: ICML, AAAI, IJCAI. CV: CVPR, ICCV, ECCV. DM: KDD, WSDM, ICDM. DB: SIGMOD, VLDB, ICDE. CL: ACL, EMNLP, NAACL. IS: WWW, SIGIR, CIKM.

2014] and heterogeneous model metapath2vec [Dong et al., 2017]. In addition, a task-guided network embedding model (TaskE) [Chen and Sun, 2017] for author identification is introduced for comparison.

Evaluation Metrics.

As described in problem definition, for each author u ∈ U, papers in lu<T are treated as training data and papers in lu≥T are left for evaluation. The overlapping between lu<T and lu≥T are removed from lu≥T . We use three popular metrics, i.e., Recall@k, Precision@k and AUC, to evaluate the performance of each method. The Recall@k shows the ratio of true correlated papers returned in the top-k list, which is defined as: Rec@k = 1 |U | ∑u∈U |ˆlu≥T ⋂ lu≥T | |lu≥T | , where ˆlu≥T denotes the set of top-k papers for author u. The Precision@k reflects the accuracy of top-k papers by a method and it can be computed according to: Pre@k = 1 |U | ∑u∈U |ˆlu≥T ⋂ lu≥T | k . The AUC measures the accuracy of pairwise orders between correlated and uncorrelated papers of each author, which is formulated as: AUC = 1 |U | ∑u∈U 1 |E(u)| ∑(v,v′ )∈E(u) δ(suv > suv′), where E(u) ≡ {(v, v′ )|v ∈ lu≥T , v′ /∈ (lu<T ∪ lu≥T )} and δ is indicator function which equals 1 when the condition holds otherwise 0. The k is set to 10 and a larger Recall@k, Precision@k or AUC value means a better performance.

Experimental Settings.

All of information used for model training such as triple/pair samples in our models or the selected features in supervised learning baselines, are extracted from training data. We design two different training/test splits by setting T = 2012 and 2013. Besides, two key issues of the experiments are set as:

• Reproducibility. For the fair of comparison, we use the same feature dimension d = 128 for all content-based ranking and network embedding models. In the proposed models, the regularization parameter λ equals to 0.001. For HRIL module of TSR+, we set the number of walks (start from each author node) as 10 and the walk length as 20. Besides, we fix window size τ = 5 and hinge loss margin ξ = 0.1. In addition, we employ TensorFlow to implement the proposed model and further conduct it via NVIDIA TITAN X GPU.

• Evaluation candidates. It is time and memory consuming to extract and store features for all author-paper pairs (which amounts to over 2.7 × 1011 pairs in AMiner-F). Thus the supervised learning baselines cannot scale up to such large amount of data. To deal with this issue and reduce evaluation time, we follow the setting [Chen and Sun, 2017] that randomly samples a set of negative (uncorrelated) papers and combines it with the set of correlated papers to form a candidate set of total 200 papers for each author. In addition, we eliminate the authors who have few correlated papers (less than 3) in test set to avoid noise. After that, the average sizes of authors' correlated papers sets respectively equal to 14.7 and 12.5 in AMiner-F test data for T = 2012 and 2013, and the corresponding values in AMiner-T test data are 15.7 and 12.7. Those relative small values make the 200 candidates large enough for a convincing evaluation. The reported results are averaged over 10 experiments of such setting.

4.2 Result Comparison

The performances of all methods are reported in Table 3, where the best results are highlighted in bold and the best baselines are indicated by star notation. The last row reports the average improvements (%) of TSR+ over different baselines. Note that, the embeddings of out-of-matrix papers (published after T ) are missing in Deepwalk and metapath2vec. Accordingly we use the average of their in-matrix references' (published before T) embeddings to represent them. The main takeaways from this table are summarized as follows:

• The best content-based ranking method (word2vec+BPR) and the best network embedding models (TaskE, metapath2vec) have better average performances than the best supervised learning baseline (RandomF), which suggests that the vectorized representations generated by network embedding or content embedding are better for capturing the complicated correlations among author-paper pairs than the simple features extracted from data.
• TSR achieves better results than all baselines in most cases, showing that the joint model of deep semantic embedding and task-guided ranking is better than supervised learn-
• ing, content-based ranking and network embedding for the given task.
• TSR+ performs best in all cases for both datasets. The average improvements of TSR+ over different baselines range from 13.0% to 43.8%. In addition, TSR+ outperforms TSR, which indicates that the heterogeneous relations integrative learning module further improves TSR.

4.3 Analysis and Discussion

Parameters Sensitivity.

The hyper-parameters play important roles in TSR+, as they determine how the model will be trained. We conduct experiments to analyze the impacts of two key parameters, i.e., window size τ for model augmentation module and embedding dimension d. We investigate a specific parameter by changing its value and fixing the others. The performances of TSR+ (in terms of Rec@10 and Pre@10 on AMiner-T test data with T = 2013) on various settings of τ and d are reported in Figure 2. According to this figure:

• With the increment of τ, Rec@10 and Pre@10 increase at first since a larger window means more useful indirect correlations among authors and papers. But when τ goes beyond a certain value, the results decrease with the further increment of τ due to the possible involvement of uncorrelated noise. The best τ is around 5.
• Similar to τ, an appropriate value should be set for d such that the best representations of authors and papers are learned. The optimal value of d is around 128.

Besides d and τ, we also investigate the impacts of other hyper-parameters such as regularization parameter λ, and reveal the similar point. Therefore the certain settings of the hyper-parameters lead to the best performance of TSR+.

Performances on Different Author Groups.

As presented in introduction, an author-paper inference model should be effective for researchers with different knowledgeable levels. In order to validate the effectiveness of TSR+ on different author groups (from "cold-start" to active), we classify all authors into 6 groups (i.e., 1∼3, 4∼6, 7∼9, 10∼12, 13∼15 and >15) based on the number of observed correlated papers they have in training data, then evaluate the performance in each group. The performances (in terms of Rec@10 and Pre@10 with T = 2013) of our models and three selected baselines (i.e., TaskE, metapath2vec and word2vec+BPR) are shown by Figure 3. We can find that TSR+ consistently outperforms TSR and the baselines in all groups. The results indicate that TSR+ achieves robust performance across different author groups and is convincible for different purposes.

Case Study.

We present two case studies to show the details of TSR+'s result. Table 4 lists the top 5 ranked papers for two data mining researchers of different groups in previous discussion, i.e., Jure Leskovec (in the last group) and Yuxiao Dong (in the second group) in AMiner-T data (with T = 2013). It is easy to know that all return papers for both researchers belong to data mining or information system area. In addition, both of them will interact (cite or write) with 4 papers (among 5) after T, which shows that TSR+ performs well for researchers with different knowledgeable levels. As for papers rank at 4 for both cases, their topics or methods are quite similar to those of the researchers' correlated papers, which results wrong predictions of TSR+. Moreover, there are some top ranked papers published after T (publication year ≥ 2013, highlighted in red), indicating that TSR+ can correctly return not only previous works but also new papers.

5 RelatedWork

In the past decade, some works have devoted to academic data mining problems, such as heterogeneous bibliographic network analysis [Sun et al., 2012; Huang et al., 2016], citation recommendation [He et al., 2010; Ren et al., 2014] or collaborator recommendation [Tang et al., 2012; Li et al., 2014]. In this paper, we study the problem of author-paper correlation inference in big scholarly data.

The network embedding has attracted lots of attention in recent years. Most of embedding models [Perozzi et al., 2014; Grover and Leskovec, 2016; Dong et al., 2017] preserve the proximities among nodes by learning vectorized representations. Some of the extended studies have been applied to various applications in big scholarly data, like correlation inference [Huang et al., 2016; Chen and Sun, 2017] or node classification [Gui et al., 2016; Dong et al., 2017]. Unlike task-independent attribute or content unawareness of these models, our model TSR+ is task-specific and incorporates both semantic content and heterogeneous relations.

Besides academic data mining and network embedding, this paper is also related to pairwise ranking optimization [Rendle et al., 2009] in recommender systems, gated recurrent neural network [Cho et al., 2014] in deep learning, word and document embedding [Mikolov et al., 2013; Le and Mikolov, 2014] in natural language processing, etc.

6 Conclusion and FutureWork

In this paper, we propose the author-paper correlation inference problem in big scholarly data, and design a model TSR+ to solve it. The model performs joint optimization of GRU-based content encoding and task-guided ranking, and is further augmented by a heterogeneous relations integrative learning module. The extensive experiments on the well known AMiner data demonstrate that TSR+ achieves significant better performance, comparing to a number of baselines. Some potential future work includes: (1) TSR+ can be extended by using more context information like publication venue of paper; (2) the dynamics of authors' embeddings should be considered for the task since authors keep publishing new papers, and citing more papers.

Acknowledgments

We would like to thank Yuxiao Dong for suggestions. This work is supported by the Army Research Laboratory under Cooperative Agreement Number W911NF-09-2-0053 and the National Science Foundation (NSF) grant IIS-1447795. This work is partially supported by King Abdullah University of Science and Technology (KAUST).