Patent Citation Dynamics Modeling via Multi-Attention Recurrent Networks

1 Introduction

Patents are a direct outcome of research and development progress from industry and as such can serve as a signal for the direction of technological advances. Of particular interest are a patent's so-called forward citations: citations to a patent from future patents. Because new patents must cite related works and those citations are vetted by the United States Patent and Trademark Office (USPTO), forward citations serve as a marker of the importance of a patent. As a research topic, they have been used because: (1) they mirror global, cumulative inventive progress [Von Wartburg et al., 2005]; (2) they link contributors and patents associated with the technology development cycle; (3) they can discover emerging technologies at an early stage [Lee et al., 2018]; (4) they assist the measurement of patent quality [Bessen, 2008];

(5) they contain information required for technology impact analyses [Jang et al., 2017]. In today's competitive business environment, citation forecasting is a field of growing importance. By assessing the long-term potential of new technologies, such forecasts can be instrumental for decision making and risk assessment. Despite its importance as a proxy and metric for many fields of technological impact study, forecasting an individual patent's citations over time is a difficult task.

The crux of current practice in this area is to treat the forward citation chain of a patent as a temporal point process, governed by a conditional intensity function whose parameters can be learned from observations of already received citations. Conventional methods for modeling temporal point processes focus on modulating a specific parametric form of the intensity function under strong assumptions about the generative process. For example, a body of studies on predicting "resharing" behaviors, such as forward citations of papers and patents [Xiao et al., 2016; Liu et al., 2017] and retweets [Mishra et al., 2016], built their models based on the Hawkes process [Hawkes, 1971]. In the Hawkes process, the intensity function grows by a certain value whenever a new point arrives and decreases back towards background intensity. These approaches are limited by the assumption that functional forms of conditional intensity will capture the complicated dependencies among historical arrivals in real datasets. Another limit is that, with the Hawkes process, the probability density function, in the form of maximum likelihood estimation, is computed by integrating the conditional intensity function with respect to time. Thus, due to the complication of this computation, the choice of available decay kernels is limited. Throughout the literature, commonly used kernel decay functions are variants of power-low functions [Helmstetter and Sornette, 2002; Mishra et al., 2016], exponential functions [Xiao et al., 2016; Filimonov and Sornette, 2015; Liu et al., 2017] and Reyleigh functions [Wallinga and Teunis, 2004].

More recent approaches [Du et al., 2016; Wang et al., 2017; Xiao et al., 2017] attempt to employ recurrent neural networks (RNN) to encode inherent dependency structures over historical points. This removes the assumptions that the generative process can be represented via a base intensity and a decay function. RNN-based temporal point processes have been shown not only to capture the intensity structure of synthetic data (simulated by typical point process models such as Hawkes and self-correcting) [Isham and Westcott, 1979], but also to outperform conventional methods on real world datasets. There are, however, three drawbacks to exisitng RNN-based methods. First, in order to simplify integration, intensity functions are usually represented as an exponential function of the current hidden state of RNN units. In the training stage, this can easily cause numerical instability when calculating the density function for maximum likelihood estimation. And in the inference stage, this is also a computation bottleneck. Second, the goal of these methods is to predict the arrival of the next point but for patent citations it is more important to generate the entire sequence of subsequent points. Third, none of these methods considers the interaction of multiple historical sequences.

In this paper, we propose a sequence-to-sequence model for patent citation forecasting. This model uses multiple historical sequences: one each for the patent itself, its assignee, and its inventor. These three are encoded separately by RNNs and a decoder generates the citation forecast sequence based on an attention-of-attention mechanism which modulates the intervening dependency of the sequences embedded by the encoders. Furthermore, instead of explicitly designing the intensity function, we use nonlinear point-wise feed-forward networks to generate the prediction directly. Specifically, the contributions and highlights of this paper are:

• Formulating a sequence-to-sequence framework to jointly model patent citation arrival time and patent category information by learning a representation of observed historical sequences for the patent itself, its associated assignees, and related inventors.
• Designing an attention-of-attention mechanism which empowers the decoder to tackle the intervening dependency of these three historical sequences, which together contribute to the forward citation forecasting.
• Conducting comprehensive experiments on a large scale dataset collected from the USPTO to demonstrate that our model has consistently better performance for forecasting both the categories and the timestamps of new citations.

2 ProblemFormulation

For each target patent p, we compile three associated timedependent sequences. First, for the target patent itself, we collect a sequence of the timestamp and category for existing forward citations Sp = {(t〈1〉 p ,m〈1〉 p ), . . . , (t〈k〉 p ,m〈k〉 p ), . . .} where t〈k〉 p and m〈k〉 p refer to the time and the category of the forward citation, respectively. Second, we create a citation chain for the assignee who owns the target patent Sa = {t〈1〉 a , . . . , t〈k〉 a , . . .}. This chain includes patents that cite any patent owned by the assignee. Likewise, we compile the sequence of citations of all patents invented by the inventor of the target patent Sa = {t〈1〉 a , . . . , t〈k〉 a , . . .}. The key idea of using these three sequences is that, in practice, we expect forecasts to be made with a short observation window which means that the chain of forward citations for the target patent may not have sufficient records for training.

Given the input data as described above, our problem is as follows: for a target patent p, taking the first n points in Sp as observations, as well the historical citation records for its associated assignees {t〈k〉 a |t〈k〉 a < t〈n〉 p } and inventors {t〈k〉 v |t〈k〉 v < t〈n〉 p }, can we predict at what time this target patent will be cited by a new patent and to which category that new patent will belong? The question breaks down into two subproblems:

1. Predict the next citation (t〈n+1〉 p , m〈n+1〉 p ) given the observed citation sequences for the patent, its assignee, and its inventor before time t〈n+1〉 p ,
2. Predict the next l citations {(t〈n+l〉 p , m〈n+l〉 p )} given the historical citation sequences of the patent, its assignee, and its inventor before time t〈n+1〉 p .

The first subproblem implies a many-to-many recurrent neural net where the length of the input and output sequences matches the number of recurrent steps. This first subproblem is easier to solve because at each prediction k all previous historical records before timestamp t〈k〉 are available. The second subproblem is more meaningful for patent citation forecasting because it requires the model to make the next l predictions after timestamp t〈k〉 based only on the observations from t〈1〉 to t〈k〉 . Our proposed model is focused on the second subproblem, and consequently can also handle the first subproblem when l = 1.

3 Models

In this section, we present our proposed model, PC-RNN, which tackles the problem of forecasting the next l citations of a target patent. First, we show an overview of the design of the proposed framework. Then we detail the RNN architecture used on the source-side and the target-side of our model. Next we introduce the attention-of-attention mechanism which plays an important role in fusing multiple encoded sequences from the source-side for generating the predicted citation sequence. Finally, we describe the training procedure which unifies the distinct components of the proposed framework.

3.1 Model Overview

The chain of forward citations for a patent can be modeled as a marked temporal point process where the joint distribution of all citations received for a target paper p can be described as the joint density

where H〈k〉 denotes all the historical information associated with the target patent at timestamp k, t〈k〉 p is the timestamp when the k-th citation arrives, and m〈k〉 p is the category of the arriving citation. Because our primary use case is to make predictions in the early stages of a patent's life cycle, we compensate for having a short observation window by incorporating citation sequences for the assignees and inventors associated with the target patent. In particular, taking as inputs

(1) a time-dependent citation sequence of n observations of the target patent ζp = {(t〈1〉 p ,m〈1〉 p ), . . . , (t〈n〉 p ,m〈n〉 p )}, (2) records of citations received by associated assignees ζa = {t〈k〉 a |t〈k〉 a < t〈1〉 p }, and (3) records of citations received by associated inventors ζv = {t〈k〉 v |t〈k〉 v < t〈1〉 p }, the task of the proposed model is to learn an optimal distribution on observations:

where ζp, ζa, and ζv together compose the observed historical information.

Generally speaking, Eq. 2 transforms the three given sequences into another sequence. Thus, we are guided by the sequence-to-sequence paradigm [Sutskever et al., 2014] to design the proposed framework. In the overview of proposed model depicted in Fig. 1, the module that handles input sequences is called source-side and the module that makes predictions is called target-side. On the source-side, three RNNs act as encoders to separately encode the observations ζp, ζa, ζv which align at timestamp t〈1〉 p when the first citation of target patent arrives. In Fig. 1, three input sequences ζp, ζa, ζv are managed by "Encoder #1", "Encoder #2", and "Encoder #3," respectively. On the target-side, the decoder recurrently forecasts the time at which a new patent will cite the target patent and in which category that new patent will be. The target-side consists of three sublayers: a RNN layer, an attention layer, and a prediction layer. In particular, the RNN layer decodes the patent citation sequence from the source-side. The hidden state of the RNN layer enters the attention layer where an attentional hidden state is computed by fusing all the information embedded by the encoders. Finally, the prediction layer makes category and timestamp predictions.

3.2 Ingredients of PC-RNN

The three encoder RNNs separately handle the observed sequences ζp, ζa, ζv by compiling the inputs into an intermediate fixed-dimensional hidden representation. As an example, we describe the patent sequence encoder to illustrate the computation process. For the patent sequence, at the k-th observation of ζp, the category information m〈k〉 p of the citing patent is first projected into a dense representation through an embedding layer and then fed into the RNN along with temporal features:

where t〈k〉 p represents the temporal features (e.g., t〈k〉 p - t〈k-1〉 p ), m〈k〉 p is a dense representation of the category information, h〈k-1〉 p is a dp-dimensional hidden state from the last step which embedded historical information, and σp is the activation function. For clarity, we use the vanilla RNN in Eq. 3 to illustrate the computation process. In practice, we use multi-layered bidirectional LSTMs [Hochreiter and Schmidhuber, 1997] which can capture long range temporal dependencies in sequences. In this case, at each step k the output of the encoder is the summation of the hidden state of LSTMs from two directions: o〈k〉 p = -→h 〈k〉 p + ←- h 〈k〉 p . The assignee sequence encoder for ζa and the inventor sequence encoder for ζv share similar computational processes with the target patent sequence encoder except that there is no category information to embed. The workflow of patent, inventor, and assignee encoders is depicted in Fig. 1 as "Encoder #1", "Encoder #2", and "Encoder #3," respectively. On the target-side, the decoder RNN employs multi-layered unidirection LSTMs, which adds the restriction that the input point at each step is always the output from the last step.

Attention Layer

Theoretically, the decoder layer generates a citation sequence by recurrently maximizing the conditional probability

where h〈i〉 d is the hidden state of the decoder RNN at time step i. However, an RNN layer alone cannot appropriately handle the conditional dependency over the three citation sequences ζp, ζa, ζv. Thus, we propose an attention layer on top of the RNN layer. In particular, our model includes a two-level attention mechanism [Bahdanau et al., 2014; Luong et al., 2015]. First, for each time step i in the decoder, a context vector is computed for each encoder RNN using all the encoder's outputs and the current hidden state of the decoder h〈i〉 d . Using outputs frompatent encoder as an example, the computation process is defined as:

where o〈j〉 p are encoder's outputs, αp i,j are the attention weights, g(·) is an alignment function calculating the alignment weights between position j of the encoder and the ith position of the decoder, and Vp ,Wp are parameters to learn. Of the four alignment functions provided in [Luong et al., 2015], we use the "concat" function here because, in our case, source-side encoders output multiple sequences of different dimensions than the decoder's hidden state. In our experiments, the dimension of the hidden state of the patent sequence encoder and decoder is set to 32 and is 16 for assignee and inventor sequence encoders. In this step, the decoder can learn to attend to certain units of each encoder which enables the model to learn complicated dependency structures. Next, the attention weights for each context vector are calculated, based on which weighted context vectors are concatenated as the input to the next layer:

where the calculation of β is similar to the calculation of α in Eq. 5. We perform this additional attention calculation because we want the model to dynamically learn and determine the combination of context vectors from each encoder which should enhance the flexibility of the modeled memory dependency structure. We call this two-step operation an attentionof-attention mechanism. It is depicted in the green rectangle labeled "Decoder #2" in Fig. 1. Finally, the concatenated context vector ¯c〈i〉 is combined with hidden state h〈i〉 d to generate the attentional hidden state ¯h〈i〉 d which flows to the next prediction layer:

where φ is a non-linear activation function, and Vc is the parameter to learn. In our experiments, we use ReLU activation function.

Prediction Layer

In the prediction layer, for each position in the attentional hidden state ¯h〈i〉 d , two linear transformations are applied with a non-linear activation ϱ in between to further enhance the model's capability:

where h is one position in ¯h〈i〉 d , and wh1 ,wh2 , b1 , b2 are parameters to learn. The output of this layer is denoted by ˜h〈i〉 d . This operation was introduced in [Vaswani et al., 2017] which is equivalent to two 1-by-1 convolutions which is also called the "network in network" concept in [Lin et al., 2013]. In our experiments, ϱ is a ReLU function. Finally, we use two fully connected layers to generate predictions:

The total loss is the sum of the time prediction loss and the cross-entropy loss for the patent category prediction:

where n refers to the number of points in the sequence used as observations, l is the number of predictions to make, κ = m〈n+j〉 indexes the target category and consequently ˆm〈n+j〉 κ is the probability of predicting the correct category in prediction m〈n+j〉 , and ϑ(·) is defined as the absolute distance between ˆ t〈n+j〉 and t〈n+j〉 . We adopted the ADAM [Kingma and Ba, 2014] optimizer for training.

4 Experiments

We compare our PC-RNN experimentally to state-of-the-art methods for modeling marked temporal point processes on a large patent citation dataset collected from USPTO. The results show that PC-RNN outperforms the others at modeling citation dynamics.

4.1 Dataset Description and Experiment Setup

Dataset: Our dataset originates from the publicly accessible PatentsView 1 database which contains more than 6 million U.S. patents. According to patent law, new inventions must cite prior arts and differentiate their innovations from them. Patent citations have a high quality because citations are examined by officials before patents are granted. For each patent, we construct a forward citation chain using timestamps from the database. We remove patents with citation chains shorter than 20 or longer than 200. Long citation chains are relative rare in practice and the unbalanced sequence length may make training difficult. For each citing patent we also extract its NBER [Hall et al., 2001] category information. As a result, each patent has one main category and one subcategory. In total, there are 7 main categories and 37 subcategories in the entire patent database. For each patent, we also retrieved the associated assignees and inventors from the database and compiled the related citation chains for each. For patents with multiple assignees or inventors, we selected the longest available chain to provide the most information. Finally, having assembled the dataset, we sampled 15,000 sequences of which 12,000 sequences are used for the training set and the remaining 3,000 sequences are the test set. Our dataset and code is publicly available for download. 2

1 http://www.patentsview.org/download/

Metrics: Following the similar procedure in [Du et al., 2016; Wang et al., 2017; Xiao et al., 2017], two metrics are used to compare the performance of different tasks. For event time prediction, we used Mean Absolute Error (MAE). For both main category and subcategory prediction, we used the Accuracy metric.

Comparison Models: State-of-the-art RNN based prediction methods can be generally categorized into two classes. (1) Intensity-based RNNs represents an observed sequence with the hidden state of a recurrent neural network. They then formulate an intensity function conditioned on the hidden state to make predictions as in a typical temporal point process. (2) End-to-end RNNs avoid explicitly designated conditional intensity functions, and instead represent the mapping from input sequences to predictions implicitly within the network structure. Our method belongs to the second class. In our experiments, we compare our model against three other major peer recurrent temporal point process models of both classes, each of which predicts both the timestamp and category of arrival points using observed sequences as input:

• RMTPP [Du et al., 2016]: Recurrent marked temporal point process (RMTPP) is an intensity-based RNN model for general point process analysis which is able to predict both the timestamp and the type of point in an event sequence. RMTPP is not a sequence-to-sequence model, however, by directing the k-th output to the (k + 1)-th input, the model can be used as a sequence generator. RMTPP only models one sequence. So, in the experiment, we feed only the patent sequence to the model as input.
• CYAN-RNN [Wang et al., 2017]: CYAN-RNN is an intensity-based RNN model which models a general resharing dependence in social media datasets. It can forecast the time and user of the next resharing behavior. Similarly to RMTPP, by forcing the k-th output to the

2 https://github.com/TaoranJ/PC-RNN

(k+1)-th input, this model is used as a sequence generator for patent citation prediction. For this experiment, we use only patent citation sequences as input to generate predictions.

• IntensityRNN [Xiao et al., 2017]: In IntensityRNN, a time-dependent event arrival sequence and an evenly distributed background time series work together to generate predictions for the next arrival event. In experiments, the evenly distributed background time series is simulated with assignee citations which usually have a stable citation pattern. IntensityRNN is an end-to-end RNN model.

4.2 Results and Discussion

The motivation for our model is to use the historical forward citations of a patent's assignees and inventors to boost the prediction performance since the observation window of the target patent is limited. Therefore, our experiment tests observation windows of different lengths. In particular, in four experiments, we use 80%, 50%, 30% and 10% of the patent sequence as observations. As the observation window becomes shorter, the predictive task becomes more difficult but also more salient for real world applications. In each experiment, to fully examine each model's performance at category prediction, each model runs the following two tasks separately:

• prediction of the timestamp and the NBER main category of the newly arrived citation.
• prediction of the timestamp and the NBER subcategory of the newly arrived citation.

Experiments are conducted with the test dataset which is separate from the training set. Results are reported in Table 1. We consider RMTPP and CYAN-RNN to be "intensitybased RNNs" because, instead of using recurrent neural nets, they treat hidden states as the embedded historical information from which they directly predict using an explicitly conditional intensity function. In contrast, IntensityRNNbelongs to the end-to-end RNN model because it directly makes desired predictions given input. To better understand the experimental results, we analyze the performance of each class as compared to our model.

Intensity-Based Models

Our model consistently and significantly outperforms RMTPP and CYAN-RNN for category classification in all experiments. This suggests that these two models struggle to modulate patent categories and subcategories. Similar results about weak classification performance using these two models on different datasets are reported in [Du et al., 2016; Wang et al., 2017]. For prediction of the next forward citation, our model consistently outperforms these intensitybased models. As the observation window shrinks, the performance of our model on the category classification task decreases, though still outperforms its competitors. However, the performance of our model on the time prediction task stays relatively stable. We observe that the timestamp information contained in the assignee's and inventor's citation chains could help to predict upcoming citations to the target patent. In contrast, the performance of the intensitybased models on the time prediction task decreases significantly as the available observations are reduced. We observe that the optimizations for intensity-based methods is sensitive to the number of observations, especially considering that maximum likelihood estimation is used. As a result, we conclude that including sequences associated to the main target sequences in our model helps to improve its sensitivity to the number of observations.

End-to-End Models

Our model consistently outperforms IntensityRNN on the category classification task. In particular, for main category classification, our model is 6% to 26% more accurate compared to IntensityRNN across all observation windows. This improvement is even more significant for subcategory classification, where our model has an improvement of 16% to 26%. We argue that this boost is attributed to our attentionof-attention mechanism. In our mechanism, the first attention layer allows the decoder to look back at all the hidden states for each encoder at each step and dynamically attend to only the important states in each sequence. The second attention layer empowers the model to fuse attended parts from different sequences which manages the dependencies across input sequences. In contrast, IntensityRNN relies solely on the last hidden state to carry the entire sequence's information and the two input sequences are connected only through a fully connected layer which may cause information loss. On the time prediction task, our model outperforms IntensityRNN on most test cases, with the exception of two. Considering IntensityRNN's poorer performance at the category classification task in those two tests, we suggest that IntensityRNN's loss functions focus more on time loss which impacts classification performance. In contrast, the loss function in our model is more balanced and encourages the model to pay attention to the two tasks equally.

In summary, we conclude that incorporating assignee and inventor information in the source-side encoder provides extra information which improves our model's performance at the prediction task. Additionally, as a proxy for prediction, using an end-to-end learning diagram to modulate a marked temporal point process is better than using a specific parametric form of a conditional intensity function, at least in the domain of patent citation classification. Finally, the empirical results suggest that the attention-of-attention mechanism enables the decoder to dynamically observation historical information from different input sequences and consequently enhance model performance.

5 Conclusion

In this paper, we present a patent citation dynamics model with fused attentional recurrent network. This model expands the traditional sequence-to-sequence paradigm by considering multiple sequences on the source-side: one each for the target patent, its assignee, and its inventor. We employ an attention-of-attention mechanism which encourages the decoder to attend to certain parts of each source-side sequence and then further fused the attended parts, grouping the by sequences which leverage dynamically learned weights. Our model is evaluated on on patents data collected from USPTO and experimental results demonstrate that our model can consistently outperform state-of-the-art temporal point process modeling methods at predicting time of newly arrived citations and classification of their category and subcategory.

Acknowledgments

This work is supported in part by the National Science Foundation via grants DGE-1545362 and IIS-1633363. The US Government is authorized to reproduce and distribute reprints of this work for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF, or the U.S. Government.