Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks

1 Introduction

Supervised learning is one of the fundamental tasks in machine learning (ML) and data mining. The goal is to infer a function that predicts the target given predictor variables (aka. features) as input. For example, real valued targets for regression and categorical labels for classification. It has broad applications including recommendation systems [Bayer et al., 2017; Zhao et al., 2016], online advertising [Shan et al., 2016; Juan et al., 2016], and image recognition [Zhang et al., 2017; Wang et al., 2015].

When performing supervised learning on categorical predictor variables, it is important to account for the interactions between them [He and Chua, 2017; Cheng et al., 2016]. As an example, let us consider the toy problem of predicting customers' income with three categorical variables: 1) occupation = {banker,engineer,...}, 2) level = {junior,senior}, and 3) gender = {male,female}. While junior bankers have a lower income than junior engineers, it can be the other way around for customers of senior level - senior bankers generally have a higher income than senior engineers. If a ML model assumes independence between predictor variables and ignores the interactions between them, it will fail to predict accurately, such as linear regression that associates a weight for each feature and predicts the target as the weighted sum of all features.

∗The corresponding author is Xiangnan He.

To leverage the interactions between features, one common solution is to explicitly augment a feature vector with products of features (aka. cross features), as in polynomial regression (PR) where a weight for each cross feature is also learned. However, the key problem with PR (and other similar cross feature-based solutions, such as the wide component of Wide&Deep [Cheng et al., 2016]) is that for sparse datasets where only a few cross features are observed, the parameters for unobserved cross features cannot be estimated.

To address the generalization issue of PR, factorization machines (FMs) 1 were proposed [Rendle, 2010], which parameterize the weight of a cross feature as the inner product of the embedding vectors of the constituent features. By learning an embedding vector for each feature, FM can estimate the weight for any cross feature. Owing to such generality, FM has been successfully applied to various applications, ranging from recommendation systems [Wang et al., 2017a; Chen et al., 2016] to natural language processing [Petroni et al., 2015]. Despite great promise, we argue that FM can be hindered by its modelling of all factorized interactions with the same weight. In real-world applications, different predictor variables usually have different predictive power, and not all features contain useful signal for estimating the target, such as the gender variable for predicting customers' income in the previous example. As such, the interactions with less useful features should be assigned a lower weight as they contribute less to the prediction. Nevertheless, FM lacks such capability of differentiating the importance of feature interactions, which may result in suboptimal prediction.

In this work, we improve FM by discriminating the importance of feature interactions. We devise a novel model named AFM, which utilizes the recent advance in neural network modelling - the attention mechanism [Chen et al., 2017a; 2017b] - to enable feature interactions contribute differently to the prediction. More importantly, the importance of a feature interaction is automatically learned from data without any human domain knowledge. We conduct experiments on two public benchmark datasets of context-aware prediction and personalized tag recommendation. Extensive experiments show that our use of attention on FM serves two benefits: it not only leads to better performance, but also provides insight into which feature interactions contribute more to the prediction. This greatly enhances the interpretability and transparency of FM, allowing practitioners to perform deeper analysis of its behavior.

1 In this paper, we focus on the second-order FM, which is the most effective and widely used instance of FMs.

2 FactorizationMachines

As a general ML model for supervised learning, factorization machines were originally proposed for collaborative recommendation [Rendle, 2010; Rendle et al., 2011]. Given a real valued feature vector x ∈ Rn where n denotes the number of features, FM estimates the target by modelling all interactions between each pair of features:

where w0 is the global bias, wi denotes the weight of the i-th feature, and ˆwij denotes the weight of the cross feature xixj, which is factorized as: ˆwij = vTi vj, where vi ∈ Rk denotes the embedding vector for feature i, and k denotes the size of embedding vector. Note that due to the coefficient xixj , only interactions between non-zero features are considered.

It is worth noting that FMmodels all feature interactions in the same way: first, a latent vector vi is shared in estimating all feature interactions that the i-th feature involves; second, all estimated feature interactions ˆwij have a uniform weight of 1. In practice, it is common that not all features are relevant to prediction. As an example, consider the problem of news classification with the sentence "US continues taking a leading role on foreign payment transparency". It is obvious that the words besides "foreign payment transparency" are not indicative of the topic of the (financial) news. Those interactions involving irrelevant features can be considered as noises that have no contribution to the prediction. However, FM models all possible feature interactions with the same weight, which may adversely deteriorate its generalization performance.

3 Attentional FactorizationMachines

3.1 Model

Figure 1 illustrates the neural network architecture of our proposed AFM model. For clarity purpose, we omit the linear regression part in the figure, which can be trivially incorporated. The input layer and embedding layer are the same with FM, which adopts a sparse representation for input features and embeds each non-zero feature into a dense vector. In the following, we detail the pair-wise interaction layer and the attention-based pooling layer, which are the main contribution of this paper.

Pair-wise Interaction Layer

Inspired by FM that uses inner product to model the interaction between each pair of features, we propose a new Pairwise Interaction Layer in neural network modelling. It expands m vectors to m(m - 1)/2 interacted vectors, where each interacted vector is the element-wise product of two distinct vectors to encode their interaction. Formally, let the set of non-zero features in the feature vector x be X , and the output of the embedding layer be E = {vixi}i∈X . We can then represent the output of the pair-wise interaction layer as a set of vectors:

where ⊙ denotes the element-wise product of two vectors, and Rx = {(i, j)}i∈X,j∈X,j>i for short. By defining the pair-wise interaction layer, we can express FM under the neural network architecture. To show this, we first compress fPI(E) with a sum pooling, and then use a fully connected layer to project it to the prediction score:

where p ∈ Rk and b ∈ R denote the weights and bias for the prediction layer, respectively. Clearly, by fixing p to 1 and b to 0, we can exactly recover the FM model. Note that our recent work of neural FM has proposed a Bilinear Interaction pooling operation [He and Chua, 2017], which can be seen as using a sum pooling over the pair-wise interaction layer.

Attention-based Pooling Layer

Since the attention mechanism has been introduced to neural network modelling, it has been widely used in many tasks, such as recommendation [Chen et al., 2017a], information retrieval [Xiong et al., 2017], and computer vision [Chen et al., 2017b]. The idea is to allow different parts contribute differently when compressing them to a single representation. Motivated by the drawback of FM, we propose to employ the attention mechanism on feature interactions by performing a weighted sum on the interacted vectors:

where aij is the attention score for feature interaction ˆwij, which can be interpreted as the importance of ˆwij in predicting the target. To estimate aij, an intuitive solution to directly learn it by minimizing the prediction loss, which also seems to be technically viable. However, the problem is that, for features that have never co-occurred in the training data, the attention scores of their interactions cannot be estimated. To address the generalization problem, we further parameterize the attention score with a multi-layer perceptron (MLP), which we call the attention network. The input to the attention network is the interacted vector of two features, which encodes their interaction information in the embedding space. Formally, the attention network is defined as:

where W ∈ Rt×k , b ∈ Rt , h ∈ Rt are model parameters, and t denotes the hidden layer size of the attention network, which we call attention factor. The attention scores are normalized through the softmax function, a common practice by previous work. We use the rectifier as the activation function, which empirically shows good performance.

The output of the attention-based pooling layer is a k dimensional vector, which compresses all feature interactions in the embedding space by distinguishing their importance. We then project it to the prediction score. To summarize, we give the overall formulation of AFMmodel as:

where aij has been defined in Equation (5). The model parameters are Θ = {w0, {wi}ni=1 , {vi}ni=1 , p, W, b, h}.

3.2 Learning

As AFM directly enhances FM from the perspective of data modelling, it can also be applied to a variety of prediction tasks, including regression, classification and ranking. Different objective functions should be used to tailor the AFM model learning for different tasks. For regression task where the target y(x) is a real value, a common objective function is the squared loss:

where T denotes the set of training instances. For binary classification or recommendation task with implicit feedback [He et al., 2017b], we can minimize the log loss. In this paper, we focus on the regression task and optimize the squared loss.

To optimize the objective function, we employ stochastic gradient descent (SGD) - a universal solver for neural network models. The key to implement a SGD algorithm is to obtain the derivative of the prediction model ˆ yAFM(x) w.r.t. each parameter. As most modern toolkits for deep learning have provided the functionality of automatic differentiation, such as Theano and TensorFlow, we omit the details of derivatives here.

Overfitting Prevention

Overfitting is a perpetual issue in optimizing a ML model. It is shown that FM can suffer from overfitting [Rendle et al., 2011], so the L2 regularization is an essential ingredient to prevent overfitting for FM. As AFM has a stronger representation ability than FM, it may be even easier to overfit the training data. Here we consider two techniques to prevent overfitting - dropout and L2 regularization - that have been widely used in neural network models.

The idea of dropout is randomly drop some neurons (along their connections) during training [Srivastava et al., 2014]. It is shown to be capable of preventing complex co-adaptations of neurons on training data. Since AFM models all pairwise interactions between features while not all interactions are useful, the neurons of the pair-wise interaction layer may easily co-adapt with each other and result in overfitting. As such, we employ dropout on the pair-wise interaction layer to avoid co-adaptations. Moreover, as dropout is disabled during testing and the whole network is used for prediction, dropout has another side effect of performing model averaging with smaller neural networks, which may potentially improve the performance [Srivastava et al., 2014].

For the attention network component which is a one-layer MLP, we apply L2 regularization on the weight matrix W to prevent the possible overfitting. That is, the actual objective function we optimize is:

where λ controls the regularization strength. We do not employ dropout on the attention network, as we find the joint use of dropout on both the interaction layer and attention network leads to some stability issue and degrades the performance.

4 RelatedWork

FMs [Rendle, 2010] are mainly used for supervised learning under sparse settings; for example, in situations where categorical variables are converted to sparse feature vector via one-hot encoding. Distinct from the continuous raw features found in images and audios, input features of the Web domain are mostly discrete and categorical [He and Chua, 2017]. For prediction with such sparse data, it is crucial to model the interactions between features [Shan et al., 2016]. In contrast to matrix factorization (MF) that models the interaction between two entities only [He et al., 2016b], FM is designed to be a general machine learner for modelling the interactions between any number of entities. By specifying the input feature vector, [Rendle, 2012] shows that FM can subsume many specific factorization models such as MF, parallel factor analysis, and SVD++ [Koren, 2008]. As such, FM is recognized as the most effective linear embedding method for sparse data prediction. Many variants to FM have been proposed, such as the neural FM [He and Chua, 2017] that deepens FM under the neural framework to learn high-order feature interactions, and the field-aware FM [Juan et al., 2016] that associates multiple embedding vectors for a feature to differentiate its interaction with other features of different fields.

In this work, we contribute improvements of FM by discriminating the importance of feature interactions. We are aware of a work similar to our proposal - GBFM [Cheng et al., 2014], which selects "good" features with gradient boosting and models only the interactions between good features. For interactions between selected features, GBFM sums them up with the same weight as FM does. As such, GBFM is essentially a feature selection algorithm, which is fundamentally different with our AFM that can learn the importance of each feature interaction.

Along another line, deep neural networks (aka. deep learning) are becoming increasingly popular and have recently been employed to prediction under sparse settings. Specifically, [Cheng et al., 2016] proposes Wide&Deep for App recommendation, where the Deep component is a MLP on the concatenation of feature embedding vectors to learn feature interactions; and [Shan et al., 2016] proposes DeepCross for click-through rate prediction, which applies a deep residual MLP [He et al., 2016a] to learn cross features. We point out that in these methods, feature interactions are implicitly captured by a deep neural network, rather than FM that explicitly models each interaction as the inner product of two features. As such, these deep methods are not interpretable, as the contribution of each feature interaction is unknown. By directly extending FM with the attention mechanism that learns the importance of each feature interaction, our AMF is more interpretable and empirically demonstrates superior performance over Wide&Deep and DeepCross.

5 Experiments

We conduct experiments to answer the following questions:

• RQ1 How do the key hyper-parameters of AFM (i.e., dropout on feature interactions and regularization on the attention network) impact its performance?
• RQ2 Can the attention network effectively learn the importance of feature interactions?
• RQ3 How does AFM perform as compared to the state-of-theart methods for sparse data prediction?

5.1 Experimental Settings

Datasets. We perform experiments with two public datasets: Frappe [Baltrunas et al., 2015] and MovieLens 2 [Harper and Konstan, 2015]. The Frappe dataset has been used for context-aware recommendation, which contains 96, 203 app usage logs of users under different contexts. The eight context variables are all categorical, including weather, city, daytime and so on. We convert each log (user ID, app ID and context variables) to a feature vector via one-hot encoding, obtaining 5, 382 features. The MovieLens data has been used for personalized tag recommendation, which contains 668, 953 tag applications of users on movies. We convert each tag application (user ID, movie ID and tag) to a feature vector and obtain 90, 445 features.

Evaluation Protocol. For both datasets, each log is assigned a target of value 1, meaning the user has used the app under the context or applied the tag on the movie. We randomly pair two negative samples with each log and set their target to -1. As such, the final experimental data for Frappe and MovieLens contain 288, 609 and 2, 006, 859 instances, respectively. We randomly split each dataset into three portions: 70% for training, 20% for validation, and 10% for testing. The validation set is only used for tuning hyper-parameters, and the performance comparison is done on the test set. To evaluate the performance, we adopt root mean square error (RMSE), where a lower score indicates a better performance.

Baselines. We compare AFM with the following competitive methods that are designed for sparse data prediction:

• LibFM [Rendle, 2012]. This is the official C++ implementation for FM. We choose the SGD learner as other methods are all optimized by SGD (or its variants).
• HOFM. This is the TensorFlow implementation 3 of the higher-order FM [Blondel et al., 2016]. We set the order size to 3, as the MovieLens data has only three types of predictor variables (user, item, and tag).
• Wide&Deep [Cheng et al., 2016]. We implement the method. As the structure (e.g., depth and size of each layer) of a deep neural network is difficult to be fully tuned, we use the same structure as reported in the paper. The wide part is the same as the linear regression part of FM, and the deep part is a three-layer MLP with the layer size 1024, 512 and 256.
• DeepCross [Shan et al., 2016]. We implement the method with the same structure of the original paper. It stacks 5 residual units (each unit has two layers) with hidden dimension 512, 512, 256, 128 and 64, respectively.

All models are learned by optimizing the squared loss for a fair comparison. Besides LibFM, all methods are learned by the mini-batch Adagrad. The batch size for Frappe and MovieLens is set to 128 and 4096, respectively. The embedding size is set to 256 for all methods. Without special in)

2 grouplens.org/datasets/movielens/latest

3 https://github.com/geffy/tffm

t)

)

0 100

MovieLens

AFM

FM

0.36

LibFM

RMSE(validation)

0.35

0.34

0.33

0.32

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Dropout Ratio

s

M

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Dropout Ratio

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Dropout Ratio

0.49

0.32

0.31

0.31

RM

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

RMS

0.45

RMSE(

0.32

RMS

0.33

0.44

0.31

0 0.5 1 2 4 8 16

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Dropout Ratio 0.31 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Dropout Ratio Dropout Ratio λ 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Dropout Ratio 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Dropout Ratio Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

Frappe AFM

FM

LibFM

MovieLens

0.48

AFM

0.41

0.39

LibFM

RMSE(validation)

0.37

0.35

0.33

0.31

0.47

0.46

0.45

0.44

0.43

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Dropout Ratio

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Figure 2: Validation error of AFM and FM w.r.t. different dropout

Attention Factors

ratios on the pair-wise interaction layer

Frappe

0.48

MovieLens

mention, the attention factor is also 256, same as the em-

0.35

bedding size. We carefully tuned the L2 regularization for

Frappe

0.34

AFM

0.33

FM

Frappe

AFM(train)

AFM(test)

FM(train)

LibFM

0.32

0.31

1 4 8 16 32 64 128 256

0.30

0 0.5 1 2 4 8 16

1 4 8 16 32 64 128 256

60

Epoch

0.47

0.48

Attention Factors

action layer. We set λ to 0, so that no L2 regularization is

80

0.48

used on the attention network. We also validate dropout on

0.60

0.47

0.60

0.50

0.47

our implementation of FM by removing the attention compo-

0.50

Frappe

0.60

AFM(train)

AFM(test)

FM(train)

nent of AFM. Figure 2 shows the validation error of AFM and

Frappe

AFM

FM

LibFM

AFM

0.40

0.46

FMw.r.t. different dropout ratios; the result of LibFM is also

FM

0.40

0.46

0.45

0.50

FM(test)

AFM(train)

AFM(test)

FM(train)

FM(test)

shown as a benchmark. We have the following observations:

LibFM

0.30

0.44

0.40

· By setting the dropout ratio to a proper value, both AFM

0.30

RMSE(validation)

0.30

0.20

0.10

and FM can be significantly improved. Specifically, for

0.00

RMSE

1 4 8 16 32 64 128 256

0.45

0.44

0.43

AFM, the optimal dropout ratio on Frappe and MovieLens

4 8 16 32 64 128 256

0

is 0.2 and 0.5, respectively. This verifies the usefulness of

dropout on the pair-wise interaction layer, which improves

RMSE(validation)

.5 0.6 0.7 0.8

atio

0.30

Frappe RMSE

the generalization of FM and AFM.

0 10 20 30 40 50 60 70 80 90 100

Epoch

0.60

MovieLens

· Our implementation of FM offers a better performance

0.60

0.50

0.40

0.30

than LibFM. The reasons are twofold. First, LibFM op-

0.50

timizes with the vanilla SGD, which adopts a fixed learn-

AFM(train)

AFM(test)

FM(train)

FM(test)

AFM(train)

ing rate for all parameters; while we optimize FM with

0.40

0.30

Adagrad, which adapts the learning rate for each parame-

0.20

RMSE

AFM(test)

FM(train)

FM(test)

ter based on its frequency (i.e., smaller updates for frequent

0.00 0.10 0.20 0 20 40 60 80 100 0.00 0.10 0 10 20 30 40 Epoch and larger updates for infrequent parameters). Second, LibFM prevents overfitting via L2 regularization, while we employ dropout, which can be more effective due to the model averaging effect.

• Epoch · AFM outperforms FM and LibFM by a large margin. Even when dropout is not used and the overfitting issue does exist to a certain extent, AFM achieves a performance significantly better than the optimal performance of LibFM and FM (cf. the result of dropout ratio equals to 0). This demonstrates the benefits of the attention network in learning the weight of feature interactions.

We then study whether the L2 regularization on the attention network is beneficial to AFM. The dropout ratio is set to the optimal value for each dataset, as evidenced by the previous experiment. As can be seen from Figure 3, when λ is set to a value larger than 0, AFM is improved (note that the result of

Frappe RMSE

0.20

RMSE

RMSE(validation)

FM(test)

RMSE(validation)

MovieLens Frappe MovieLens

0

0.60

0.50

0.40

0.30

0.20

10

0.10

0.00

0 0.5 1 2 4 8 16 λ 0 20 40 60 80 100 Epoch 0.00 0.10 0 10 20 30 40 Epoch FM(test) 20 30 40 50 Epoch 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0 10 20 30 40 50 60 70 80 90 100 RMSE Frappe AFM(train) AFM(test) FM(train) FM(test) 0.43 1 4 8 16 32 64 128 256 Attention Factors 0.00 0 10 20 30 40 50 60 70 80 We now focus on analyzing the impact of the attention network on AFM. The first question to answer is how to choose a proper attention factor? Figure 4 shows the validation error of AFM w.r.t. different attention factors. Note that λ has been separately tuned for each attention factor. We can observe that for both datasets, AFM's performance is rather stable across attention factors. Specifically, when the attention factor is 1, the Wmatrix becomes a vector and the attention network essentially degrades to a linear regression model with the interacted vector (i.e., vi ⊙ vj) as input features. Despite such restricted model capability of the attention component, AFM remains to be very strong and significantly improves

Epoch

over FM. This justifies the rationality of AFM's design that

estimates the importance score of a feature interaction based

50

on its interacted vector, which is the key finding of this work.

Figure 5 compares the training and test error of AFM and FM of each epoch. We observe that AFM converges faster than FM. On Frappe, both the training and test error of AFM are much lower than that of FM, indicating that AFM can better fit the data and lead to more accurate prediction. On MovieLens, although AFM achieves a slightly higher training error than FM, the lower test error shows that AFM generalizes better to unseen data.

Micro-level Analysis

Besides the improved performance, another key advantage of AFM is that it is more explainable through interpreting the attention score of each feature interaction. To demonstrate Epoch

50

0.31

0.35

0.34

0.33

0.32

0.31

RMSE

MovieLens RMS

0.31

0.30

0.10

0.00

Epoch

RMSE(validation)

RMS

0.30

0.32

0.31

0 0.5 1 2 4 8 16

λ

AFM

FM

LibFM

0 0.5 1 2 4 8 16

Figure 5: Training and test error of each epoch

Frappe Frappe this, we perform some micro-level analysis by investigating the score of each feature interaction on MovieLens.

most important. However, FM assigns the same importance

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0 20 40 60 80 100 RMSE Epoch AFM(train) AFM(test) FM(train) FM(test) 0.00 0.10 0.20 0.30 0.40 0.50 0.60 RMSE 0.30 0.31 0.32 0.33 0.34 0.35 1 4 8 16 32 64 128 256 RMSE(validation) Attention Factors AFM FM LibFM To allow a dedicated analysis on the attention network, we first fix aij to a uniform number 1/|Rx|, training the model which simulates FM. We then fix feature embeddings, training the attention network only; upon convergence, the performance is improved about 3%, which justifies the efficacy of the attention network. We then select three test examples of target value 1, showing the attention score and interaction score of each feature interaction in Table 1. We can see that among all three interactions, the item-tag interaction is the

0.48

0.47

0.46

MovieLens

0.33

0.31

Frappe Table 2: Test error and number of parameters of different methods on embedding size 256. M denotes "million".

	0.35 Frappe		MovieLens
Method	0.34 Param#	RMSE	Param#	RMSE
LibFM	1.38M	0.3385	23.24M	0.4735
HOFM	0.33 2.76M	AFM 0.3331 46.40M		0.4636
0.32 Wide&Deep 4.66M		FM LibFM 0.3246 24.69M		0.4512
RMSE(validation) DeepCross 8.93M		0.3548 25.42M		0.5130
0.31 AFM 1.45M		1 2 0.3102 23.26M		8 16	0.4325

score for all interactions, resulting in a large prediction er-

ror. By augmenting FMwith the attention network (cf. rows

FM+A), the item-tag interaction is assigned a higher impor-

tance score, and the prediction error is reduced.

AFM

FM

Table 1: The attention score*interaction score of each feature inter-

LibFM

0.45

0.44 #	Model	User-Item	User-Tag	Item-Tag	ˆ y
0.43 FM 0.33*-1.81			0.33*-2.65	0.33*4.55	0.03
1 4 8 16 32 64 128 256 1 FM+A 0.34*-1.81			0.27*-2.65	0.38*4.55	0.39
2	Attention Factors FM 0.33*-1.62		0.33*-1.00	0.33*3.32	0.24
	FM+A	0.38*-1.62	0.20*-1.00	0.42*3.32	0.56
3	FM	0.33*-1.40	0.33*-1.26	0.33*4.68	0.67
	FM+A	0.33*-1.40	0.29*-1.26	0.37*4.68	0.89

action of three test examples on MovieLens.

0 20 40 60 80 100 Epoch 6 Conclusion and FutureWork

0.20 0.30 0.40 0.50 0.60 RMSE Frappe AFM(train) AFM(test) FM(train) FM(test) We have presented a simple yet effective model AFM for supervised learning. Our AFM enhances FM by learning the importance of feature interactions with an attention network, which not only improves the representation ability but also the interpretability of a FM model. This work is orthogonal with our recent work on neural FM [He and Chua, 2017] that develops deep variants of FM for modelling high-order feature interactions, and it is the time that introduces the attention mechanism to factorization machines.

5.4 Performance Comparison (RQ3)

In this final subsection, we compare the performance of different methods on the test set. Table 2 summarizes the best performance obtained on embedding size 256 and the number of trainable parameters of each method.

• First, we see that AFM achieves the best performance among all methods. Specifically, AFM betters LibFM with a 8.6% relative improvement by using less than 0.1M additional parameters; and AFM outperforms the second best method Wide&Deep with 4.3%, while using much fewer model parameters. This demonstrates the effectiveness of AFM, which, despite being a shallow model, achieves better performance than deep learning methods.
• Second, HOFM improves over FM, which is attributed to its modelling of higher-order feature interactions. However, the slight improvements are based on the rather expensive cost of almost doubling the number of parameters, as HOFM uses a separated set of embeddings to model the feature interactions of each order. This points to a promising direction of future research - devising more effective methods for capturing higher-order feature interactions.

0.00 0.10 0 20 40 60 80 100 Epoch In future, we will explore deep version for AFM by stacking multiple non-linear layers above the attention-based pooling layer and see whether it can further improve the performance. As AFM has a relatively high complexity quadratic to the number of non-zero features, we will consider improving its learning efficiency, for example by using learning to hash [Zhang et al., 2016b; Shen et al., 2015] and data sampling [Wang et al., 2017b] techniques. Another promising direction is to develop FM variants for semisupervised and multi-view learning, for example by incorporating the widely used graph Laplacian [He et al., 2017a; Wang et al., 2016] and co-regularization designs [He et al., 2014; Yang et al., 2015]. Lastly, we will explore AFM on modelling other types of data for different applications, such as texts for question answering [Zhao et al., 2015] and more semantic-rich multi-media content [Zhang et al., 2016a; Yang et al., 2014].

Acknowledgment The work is supported by the National Natural Science Foundation of China under Grant No.U1611461 and No.61572431, Key Research and Development Plan of Zhejiang Province under Grant No.2015C01027, Zhejiang Natural Science Foundation under Grant No.LZ17F020001. NExT research is supported by the National Research Foundation, Prime Minister's office, Singapore under its IRC@SG Funding Initiative.

RMS