Hi-Fi Ark: Deep User Representation via High-Fidelity Archive Network

1 Introduction

The flourish of deep learning techniques has greatly promoted the progress of recommendation systems. One of the most notable practices is user representation, where deep neural networks are employed to extract users' stable behavioral patterns in the long run. For such kind of applications, neural networks work as an encoding function of user history, which encodes user's underlying preference into compact vectors. Conventionally, user representations are well generated in the offline stage, making it independent of specific candidate item and incapable of making adaption for the recommendation context. Because of its simplicity, the candidate-independent representation is widely adopted in industry. However, without referencing to specific recommendation context, the candidate-independent representation will probably be too generalized to reflect user's preference from the interested aspect, which inevitably impairs the recommendation performance.

Recently, more advanced approaches are inspired by the popularity of attention mechanisms [Zhou et al.2018b, Zhou et al.2018a,Wang et al.2018]. By attentively aggregating user history w.r.t. the given candidate, user preference can be better captured from the perspective of interest. However, such kind of user representation has to be conducted at runtime, (i.e., everytime a specific candidate needs to be recommended for the user,) and it requires the access to the user's entire history. As a result, it will repetitively incur a series of timeconsuming data retrieval operations. (For example, the access to a user's viewed ads involves the selection of her previous ads-clicks from activity logs, followed by the join with tables of ads' contents.) Considering that recommendations must be made in real-time for many applications, like news feed and online advertising, the deployment feasibility of such methods will be severely limited. In fact, it remains a challenging problem to have user represented with both superior recommendation efficacy and deployment feasibility.

To address this challenge, a novel user representation framework, High-Fidelity Archive Network (Hi-Fi Ark) is proposed, which makes the best of both worlds. Instead of allocating each user with one unique representation, Hi-Fi Ark compresses user history into a small constant number of "archives" in the offline stage (each of which is a vector). With this mild augmentation in space and judicious information extraction, it is expected that user preference can be comprehensively preserved with the highly compact archives. While deployed in the online service, these archives will be attentively aggregated w.r.t. the candidate item such that user preference can be fully captured from the interested perspective. To realize the above ambition, technical designs are carried out in the following ways.

First of all, one of the major deficiencies about the candidate-independent representation is that user history is summarized in a fixed and monotonous fashion, thus incapable of reflecting user preference comprehensively. In HiFi Ark, the multi-head attentive pooling mechanism is employed, which encodes user history into multiple complementary vectors. Each of these vectors is deployed to reflect user's behavioral patterns from a specific aspect; there-

Hi-Fi Ark: Hi-Fi Archive Network for Almost Lossless User Representation

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

fore, user preference can be comprehensively preserved by such compact archives. However, the direct enforcement of multi-head attentive pooling may result in incompatibility and redundancy between different heads, which will severely limit their joint representation capability. Thus in the second place, the coordinated mechanism, namely orthogonal multi-head attentive pooling, is developed. On the one hand, the pooling heads are initially set as the orthogonal basis of the items' vectorization space. It is demonstrated that such a deployment of pooling heads is desirable, as the principle of user's behavioral patterns can be fully captured with it. On the other hand, to eliminate the redundancy from subsequent operations, an auxiliary regularization function is devised, which guarantees the pooling heads' complementarity all the way through the training process. Last but not least, given that user's historical items may not be precisely represented due to insufficient feature or inaccurate encoder, considerable noise might be introduced when representations are summarized from user history. To address this problem, the intra-correlation about user history is explicitly leveraged via self-attention. By aggregating each item's representation with those relevant in semantics, essences of user behaviors can be extracted more robustly.

Major contributions of this work are summarized into the following three points.

• A novel user representation framework, Hi-Fi Ark, is proposed in this work. By judiciously summarizing user history into compact archives, and attentively aggregating these archives w.r.t. the presented candidate, user's specific preference can be precisely captured without expensive access to the entire history.
• With the joint employment of orthogonal multi-head attentive pooling and self-attention, user preference can be captured comprehensively and robustly by the archives.
• Extensive empirical studies are performed with three real-world datasets on news recommendation, online advertising, and e-commerce, respectively. It is demonstrated that Hi-Fi Ark outperforms the well-recognized baseline methods remarkably and consistently, thereby verifying the effectiveness of our proposed methods.

The rest of this paper is organized as follows. The overview of Hi-Fi Ark and its detailed analysis are presented in Section 2. Then empirical studies of Hi-Fi Ark are conducted in Section 3, followed by the related work discussed in Section 4. Finally, conclusion of this work is made in Section 5.

2 Hi-Fi Archive Network

In this section, we will get started with the preliminaries and architecture of Hi-Fi Ark; then detailed analysis will be made for each of its components.

2.1 Preliminaries and Architecture

Hi-Fi Ark works with four basic steps: item vectorization, archive generation, archives' aggregation w.r.t. candidate and CTR prediction, whose pipeline is illustrated as Figure 1. First of all, user's historical items (e.g., news clicks) Vu are mapped from raw features into continuous vectors Θu , where items' semantics are encoded 1 . Secondly, the vectorized items Θu are compressed into a small set of vectors Bu , called archives. These archives serve as flexible candidateindependent representations for the user in the offline stage. Thirdly, given a candidate item ´ v for recommendation, the archives are attentively aggregated w.r.t. the candidate's encoded vector θ´ v, where user representation relevant to the candidate, φu´ v , is produced. Finally, φu´ v and θ´ v are jointly processed by a specific feed forward function (e.g., multi-layer perception), where u's click through rate about ´ v is obtained.

To distinguish from the original way of candidate-relevant user representation, which works directly with user's entire history, definitions for the Original Candidate-relevant Representation (OCR) and the Archived Candidate-relevant Representation (ACR) are introduced as follows.

Definition 1 (OCR, ACR) Given vectorized items of user's entire history Θu , archives Bu , and a candidate item ´ v, OCR, denoted as ˆ φu´ v , is generated as the attentive aggregation of u's entire history Θu w.r.t. ´ v; meanwhile ACR, denoted as φu´ v , is the attentive aggregation of the archives Bu w.r.t. ´ v.

1 Mapping functions are individualized for different data, concrete implementations are introduced in Section 3.

It seems that OCR will always be a more precise reflection for the user's preference towards ´ v, as it is made on top of user's entire history. However, with judicious extraction of user's informative behavioral patterns via Hi-Fi Ark, comparable performance can be achieved by the resulted ACR.

2.2 Orthogonal Multi-head Attentive Pooling

Each user's vectorized items are to be summarized with k archives through Orthogonal Multi-head Attentive Pooling (OMAP). In this place, a set of k vectors Λ = {λ1, ..., λk}, namely pooling heads, are introduced. With each head λi, user u's vectorized items are attentively aggregated into the archive βu i via the following operations:

It is apparent that each pooling head summarizes user history from a unique perspective, and an increasing number of pooling heads tend to have a more comprehensive view of the user. However, it is not a plausible choice to make use of multiple heads casually. On the one hand, because of the underlying redundancy and incompatibility, it is unlikely for a small number pooling heads to get user represented comprehensively; on the other hand, a huge number of pooling heads, despite a more comprehensive view, require much more space to keep the archives, making it expensive and awkward for online deployment. To have users comprehensively represented in a compact way, the following mechanisms are developed on top of the fundamental multi-head attentive pooling.

Pooling Heads Installation. As is discussed, the efficacy of multi-head attentive pooling is greatly limited by its underlying redundancy and incompatibility. To overcome such a problem, the pooling heads are deployed w.r.t. the following pair of intuitions.

• The pooling heads should be distinguished from each other so that user history can be summarized from different perspectives; i.e., for each pair of λi, λj ∈ Λ, it is desired that the similarity between them sim(λi, λj) can be minimized.
• The pooling heads have to be deployed w.r.t. the vector distribution of the whole items ˆΘ 2 (i.e., the items ever consumed by the users), such that the principle of user information can be fully preserved.

With both considerations, the pooling heads are initialized as the orthogonal basis of ˆΘ's vector space. Particularly, such pooling heads can be acquired through the following optimization problem,

where θv is a d-dimension vector, andW is a d×k matrix. It is straightforward that the above problem is convex and fully differentiable, where the optimal solutionW∗ can be derived

2 ˆΘ can be easily acquired beforehand thanks to the popularity of unsupervised representation learning and pre-trained models.

Algorithm 1: End-to-End CTR Prediction

v;

Input: user u's historical items Vu , candidate item ´ Output: click through rate ω(u, ´ v); begin

Vectorize historical items Vu into Θu ;

Self-attention over Θu for ˙Θu via Eq. 7; Represent items as ˆΘu ←Θu + ˙Θu ; Pooling heads installation for Λ via Eq. 2; Get user archives Bu via Eq. 1, on top of Λ and ˆΘu ; Aggregate Bu w.r.t. ˆ θ´ v and get ACR φu´ v via Eq. 4; Predict CTR ω(u, ´ v) on top of φu´ v and ˆ θ´ v via Eq. 8;

via gradient descent. Finally, each pooling head λi is set to be the i-th column of W∗ :

With the above way of pooling heads installation, the resulted archives can be comparable to the original entire history, as user preference from arbitrary aspect can be well preserved. Such a property is stated as Theorem 1.

Theorem 1 Archives Bu is a near lossless compression of user history Θu , as the transformations between ACR φu´ v and OCR ˆ φu´ v are approximately invertible.

Proof 1 ACR and OCR, according to their definitions in Def. 1, are calculated as the attentive aggregations (w.r.t. ´ v) of user's archives and the entire history, respectively:

Provide that W∗ is an orthogonal basis of ˆΘ, and products between vectors (θT v θ´ v, θT v λi, βuT i θ´ v) are small enough, the following relationships can be derived:

(Notations are kind of abused here, as column vectors sets Bu and Θu can also be treated as matrices.) Based on the above relationships, it is easy to verify that there will always exist a full rank k × k matrix Ψ, which satisfies:

As a result, the following relationships can be derived:

therefore justifying the statement of Theorem 1. Notice that although dot-attention is used throughout the paper for better empirical performances, it's straightforward that the same conclusion holds for many other ways of attention, e.g., generalized dot xWyT and concatenation Wconcat(x, y)T .

The above property indicates that user's informative behavioral patterns are fully preserved on top of the proposed pooling heads installation, based on which the preference toward the candidate can be extracted with high-fidelity.

Orthogonal Pooling Heads Scattering. Following the installation operation, the pooling heads will be further incorporated into an end-to-end working pipeline (to be discussed in Section 2.4), which is trained to predict the click through rate. To ensure that redundancy is excluded from the pooling heads during such a process, function Ω is employed to restrict the changes of Λ:

(6)

In the above equation, Jk and Ik are the k-dimension allone and identity matrices, respectively, and ◦ denotes the Hadamard product. Ω is employed as a regularizer and restricted to be tiny during the training process. As a result,

orthogonality is maintained as the inner products between different pooling heads are forced to be zeros.

2.3 Self-Attentive ItemVectorization

It is desired that historical items are precisely vectorized from their raw features, so that user preference can be truthfully extracted on top of them. However, this condition doesn't always hold in reality, given that insufficient features or inaccurate encoders will introduce tremendous noise to the vectorized items. To address this problem, self-attention is employed by Hi-Fi Ark, where the intra-correlation about user history is leveraged. Instead of working with the vectorized items directly, the self-attended vector ˙ θv ( [Vaswani et al.2017]) is calculated:

As a result, each item will be aggregated with those relevant in semantics. It can be interpreted as the importance sampling for item v's underlying category, where the importance weights equal to the corresponding attention scores; with the aggregation of such samples, noise is resisted for each item and its true category can be better reflected. Following the common practice, the original vectorized item and self-attended vector are combined into the joint vector ˆ θt via residual connection, i.e., ˆ θv ← ˙ θv + θv.

2.4 End-to-End CTR Prediction

Once user's archives are well generated based on historical items, they will be deployed for the recommendation service. Presented with a candidate item ´ v, the archives will be attentively aggregated into the archived candidate-relevant representation φu´ v w.r.t. ´ v (as indicated by Eq. 4), based on which user preference is precisely reflected from the perspective of interest. Both user and item's representations, φu´ v and θ´ v, will be jointly processed by a certain type of feed forward network f so as to predict the click through rate ω(u, ´ v):

Finally, the unified end-to-end CTR prediction is shown as Alg. 1, based on which the whole model is trained to optimize the following binary classification problem,

In Eq. 9, F calculates the binary cross entropy, I(u, ´ v) represents the binary labels, and Ω is the orthogonality regularizer defined in Eq. 6. Positive instances in {U, ´V} are collected from the ground-truth clicks, while the negative ones are randomly sampled.

3 Empirical Studies

3.1 Experiment Settings

The following three datasets on news recommendation, online advertising and e-commerce are adopted for evaluation, whose statistics are shown in Table 1.

News. A total of 5-week news clicks are provided by MSN News from the EN-US market 3 . The dataset includes: 1) click relationships between users and news, 2) news titles, 3) news categories (e.g, sports, finance). Samples within the first four weeks are used for training, while those in the last week are used for testing. Titles are used as raw features, with 1D CNN employed as text encoder. The text encoder is pretrained for the classification of news category.

Ads. A total of one-week Ads clicks are offered by Bing Ads [Parsana et al.2018], which includes: 1) click relationships between users and URLs, 2) titles of URLs. The titles have already been mapped into 300-dimension vectors with a well pretrained DSSM model [Huang et al.2013] over massive corpus.

E-Commerce. This dataset 4 contains users' shopping behaviors on Amazon (the ratings-only dataset). All the purchased items are treated as positive cases, while negative cases are randomly sampled from the non-purchased ones. In contrast to other dataset, items in this dataset are purely represented by an unique ID, which is to be vectorized via a cold-started embedding matrix. As a result, we are able to evaluate the boundary case where recommendations have to be made with highly limited features.

The following baseline methods are jointly evaluated in our empirical studies.

3 https://www.msn.com/en-us/news

4 http://jmcauley.ucsd.edu/data/amazon/

Average Pooling (AVG). User representation φu´ v is generated with the average pooling of user's historical vectorized items Θu .

1D CNN (CNN). User representation φu´ v is generated with a 1D convolutional neural network followed by the average pooling operation over Θu .

Multi-headAttention (MUL). User archives are generated with multi-head attentive pooling at first as Hi-Fi Ark, but integrated into one unified vector with a kd×d mapping matrix, which follows [Vaswani et al.2017].

Single-head Attentive Pooling (SNG). A special case of multi-head attentive pooling, which only one single head is employed; thus, only one share of archive is generated.

KVMemoryNetwork (MEM). External vectors known as keys are introduced, with which user history is summarized (written) and attentively aggregated (read) [Miller et al.2016].

Direct Attention (ATT). User representation is generated by attending the entire history Θu w.r.t. the candidate, which is the mechanism in [Zhou et al.2018b,Wang et al.2018].

Because of limited space, Hi-Fi Ark is mainly evaluated for two variations with 3 pooling heads: the basic version (ARK-DE), with only multi-head attentive pooling activated, but OMAP and self-attention disabled; and the full version (ARK-EN), where all modules are enabled. Meanwhile, effects of different number of heads and each functional component are also studied. Notice that our method is for nonsequential data, thus those for temporal recommendation, e.g., Time LSTM [Zhu et al.2017], are not within discussion. We focus on evaluating the effectiveness of user representation, where a simple dot operation is used for CTR calculation (no obvious difference in performance with other simple alternatives, like MLP). However, sophisticated models, e.g., DeepFM, can be seamlessly combined. Finally, We evaluate the performance in terms of AUC scores and NDCG@(5, 10).

3.2 Experiment Analysis

The major experimental results are demonstrated in Table 2, where different methods are jointly compared over all three datasets. It can be clearly observed that ARK-EN-3 outperforms other methods on News and E-Commerce; meanwhile, it is merely second to ATT on Ads by a comparatively small margin. In our experiment, ATT is the only method, which makes recommendation on top of user's entire history (the others are based on summarization of user history). Intuitively, it should have made the best performances in all circumstances. However, we have a somewhat unexpected observation: as ARK-EN-3 beats ATT on both News and ECommerce, moreover, ARK-EN-3's performance is way better than ATT on E-Commerce. Much of this phenomenon is resulted from the differences in feature granularity. Particularly, ATT heavily relies on fine-grained features, as it will attend every single item in history to identify the relevant context to a query; ARK-*, however, is less sensitive to the feature granularity of individual items, as it works with summarization of all items in user history. Both Ads and News datasets make uses of items' texts as their raw features, but texts of Ads are encoded with a powerful DSSM model [Huang et al.2013] well-trained over massive corpus, which characterizes items much more precisely. On the other hand, only item IDs are available in the E-Commerce, where no explicit semantic information is incorporated, which leads to much weaker characterization for the items in contrast to the others. Such properties align exactly with their demonstrated performances in the experiment, which confirms our analysis. At the same time, we can also observe that ARKEN-3 is consistently better than ARK-DE-3, which indicates that OMAP and self-attention are necessary for improving the efficacy of multi-head attentive pooling. A more detailed analysis is to be made in the next part.

Effects of different components and number of heads are studied with the News dataset. Results on other datasets are omitted due to similar observations and space limit.

Componential Analysis. Componential effect is studied (in Table 3-A), with every single component, pooling heads installation (-IN), orthogonal pooling heads scattering (-OR) and self-attention (-SL), added on top of the multi-head attentive pooling. Meanwhile, the basic (-DE) and full (-EN) versions are also included for comparison. It can be observed that all the components are playing positive roles, as consistent improvements are achieved over the basic ARK-DE-3.

Hi-Fi Ark's Implementation is available at https://github.com /xyyimian/Hifi-Ark/

Moreover, the combination of all components, i.e., ARK-EN3, outperforms all the other baselines. Such a phenomenon is within our expectation, that different components work from complementary perspectives, thus magnifying the effect of every single component when integrated as a whole.

Effect of #heads. Effect of different number heads is studied with ARK-EN-*, where the pooling heads are set to be 3, 5 and 10, respectively. Besides, the special case of singlehead attentive pooling (SNG) is adopted for comparison. According to the results in Table 3-B, the employment of multiple heads does bring a remarkable leap forward, as significant improvements are achieved over SNG. However, having even more heads will not continuously benefit the efficacy, as ARK-EN-5&10 show almost no increment over ARK-EN3. Recall that the adoption of multiple heads is to ensure the comprehensive summarization of user history; and it is found in our experiment that a rank-3 setting is already capable of reconstructing all the vectorized items in the News dataset with ignorable loss (i.e., the solution of Eq. 2). Thus, there's no wonder to see the efficacy of heaving merely 3 heads. Actually, the reconstruct loss can be used as an important reference for the selection of heads number in practice.

4 RelatedWork

In this section, relate work is analyzed from two aspects: deep learning based recommendation and deep user representation.

Deep Learning-based Recommendation. Deep learningbased recommendation systems have become popular in recent years. Thanks to its powerful feature representation capability, deep learning techniques were firstly employed in recommendation system so as to introduce diverse sources of information; e.g., convolutional neural networks were applied for taking advantage of visual information [He and McAuley2016], denoised auto encoder [Zhang et al.2016] and recurrent neural networks [Bansal et al.2016] were adopted for exploiting textual information, and network embedding [Zhang et al.2016] was used to incorporate relational information. Meanwhile, deep learning is used to learn combinatorial features [Cheng et al.2016,Guo et al.2017,Lian et al.2018,Wang et al.2017a], where meaningful feature combinations are selected as discriminative patterns via deep neural networks. Finally, deep learning is also adapted as alternatives to other classical algorithms, such as matrix factorization [He et al.2017,Xue et al.2017].

Deep User Representation. Analogous to the representation learning of visual and textual information, deep learning is widely used to model user behavior, whose compact and informative representation will greatly benefit the efficacy of downstream recommendation. Roughly speaking, user representation can be generated in two ways: the candidate-independent approach and the candidate-relevant approach. Particularly, user representation can be generated in the offline stage purely based on user history [Elkahky et al.2015, Covington et al.2016], which will be directly deployed for online recommendation services. Such kind of user representation is independent of specific candidate item, and it is widely adopted in real-world industry owing to its simplicity. On the other hand, user representation may also be generated at runtime [Zhou et al.2018b, Zhou et al.2018a, Wang et al.2018], by attentively aggregating user history w.r.t. specific candidate. It is found that better recommendation performances can be achieved with the attentive aggregation of user history, where similar conclusions have also been reported in other areas like Question Answering and Semantic Matching [Wang et al.2017b,Santos et al.2016,Lin et al.2017]. Unfortunately, it is not always possible to meet the requirement of runtime access to users' entire history, making it inapplicable for many real-world scenarios. Recent works [Zhang et al.2017, Chen et al.2018] also make use of memory networks [Sukhbaatar et al.2015,Miller et al.2016] to model user's behaviors. The memory networks are structurally close to Hi-Fi Ark, as multiple memory units are introduced for the attentive aggregation of specific queries. However, little effort is made previously on preserving user's comprehensive preference with compact vectors. As is theoretically analyzed and empirically validated in our work, simple augmentation in space is far from enough for such a purpose.

5 Conclusion

A novel user representation framework, Hi-Fi Ark, is proposed in this work. With the joint employment of orthogonal multi-head attentive pooling and self-attention, user history is comprehensively summarized into compact archives; and by attentively referencing to each candidate item, user preference can be precisely captured from the perspective of interest. Thus, superior recommendation efficacy comes along with strong feasibility in Hi-Fi Ark. Extensive experiments are conducted over various real-world datasets, whose results demonstrate the effectiveness of our proposed approaches.

Acknowledgements

The authors would like to thank Microsoft News for providing technical support and data in the experiments, and JiunHung Chen and Ying Qiao for their support and discussions.