Spotting Collective Behaviour of Online Frauds in Customer Reviews

1 Introduction

Nowadays, online reviews are becoming highly important for customers to take any purchase-related decisions. Driven by the immense financial profits from product reviews, several blackmarket syndicates facilitate to post deceptive reviews to promote/demote certain products. Groups of such fraud reviewers are hired to take complete control of the sentiment about products. Collective behaviour is therefore more subtle than individual behaviour. At individual level, activities might be normal; however, at the group level they might be substantively different from the normal behavior. Moreover, it is not possible to understand the actual dynamics of a group by aggregating the behaviour of its members due to the complicated, multi-faceted, and evolving nature of inter-personal dynamics. Spammers in such collusive groups also adopt intelligent strategies (such as paraphrasing reviews of each other, employing a subset of their resources to one product, etc.) to evade detection.

∗Contact Author

Previous studies mostly focused on individual-level fraud detection [Lim et al.2010, Fei et al.2013,Akoglu et al.2013]. Few other studies which realized the detrimental effect of such collective activities detected groups simply based on Frequent Itemset Mining (FIM) [Mukherjee et al.2012, Xu et al.2013,Allahbakhsh et al.2013]. They thus focused more on ranking fraud groups, paying less attention to judge the quality of the detected groups.

[Wang et al.2016] pointed out several limitations of FIM for group detection - high computational complexity at low minimum support, absence of temporal information, unable to capture overlapping groups, prone to detect small and tighter groups, etc.

In this paper, we propose DeFrauder 1 , a novel architecture for fraud reviewer group detection. DeFrauder contributes equally to (i) the detection of potential fraud groups by incorporating several coherent behavioral signals of reviewers, and (ii) the ranking of groups based on their degree of spamicity by proposing a novel ranking strategy. Experiments on four real-world labeled datasets (two of them were prepared by us) show that DeFrauder significantly outperforms five baselines - it beats the best baseline by 11.35% higher accuracy for detecting groups, and 17.11% higher NDCG@50 for ranking groups (averaged over all datasets).

In short, our contributions are fourfold: (i) two novel datasets, (ii) novel method for reviewer group detection, (iii) novel method for ranking groups, and (iv) a comprehensive evaluation to show the superiority of DeFrauder.

2 RelatedWork

Due to the abundance of literature [Kou et al.2004] on online fraud detection, we restrict our discussion to fraud user detection on product review sites, which we deem as pertinent to this paper.

1 DeFrauder: Detecting Fraud Reviewer Groups, Code is available in [Dhawan et al.2019].

In user-level fraud detection, notable studies include [Lim et al.2010] which proposed scoring methods to measure the degree of spamicity of a reviewer based on rating behaviors; [Wang et al.2011] which developed a graph-based fraud reviewer detection model; [Fei et al.2013] which exploited burstiness in reviews to detect fake reviewers. SpEagle [Akoglu et al.2013] utilizes clues from all metadata as well as relational data and harnesses them collectively under a unified framework. ASM [Mukherjee et al.2013a] facilitates modeling spamicity as latent factor and allows to exploit various behavioral footprints of reviewers. [Wang et al.2012] argued that the dynamic behavioral signals can be captured through a heterogeneous reviewer graph by considering reviewers, reviews and products together. FraudEagle [Akoglu et al.2013] spots fraudsters and fake reviews simultaneously. [Dutta et al.2018, Chetan et al.2019] discussed about collusion in social media.

Few studies attempted to detect group-level fraud reviewer groups. GSRank [Mukherjee et al.2012] was the first method of this kind, which identifies spam reviewer groups using FIM, and ranks groups based on the relationship among groups, products, and individual reviewers. [Mukherjee et al.2012] argued that due to the scarcity of data, unsupervised method should be used to tackle this problem. Other studies largely focused on improving the ranking algorithm, ignoring the performance of the group detection method (they also used FIM for detecting groups) [Xu et al.2013, Allahbakhsh et al.2013,Mukherjee et al.2013b,Ott et al.2011]. [Xu and Zhang2015] proposed an Expectation Maximization algorithm to compute the collusive score of each group detected using FIM. [Wang et al.2016] argued that FIM tends to detect small-size and tighter groups. They proposed GSBP, a divideand-conquer algorithm which emphasizes on the topological structure of the reviewer graph. GGSpam [Wang et al.2018b] directly generates spammer groups consisting of group members, target products, and spam reviews.

We propose DeFrauder that leverages temporal pattern, content similarity and underlying graph structure to detect collusive fraud groups in review forums (see Table 1 for a comparison). Unlike others, DeFrauder contributes equally to both the components - group detection and ranking.

3 Proposed DeFrauder Framework

In this section, we explain DeFrauder which is a threestage algorithm - detecting candidate fraud groups, measuring different fraud indicators of candidate groups, and ranking groups based on the group spam score.

3.1 Group Fraud Indicators

Here we present six fraud indicators which measure the spamicity of a group [Mukherjee et al.2012, Wang et al.2018b, Wang et al.2016] by taking into account linguistic, behavioral, structural and temporal signals. All indicators are normalized to [0, 1] - larger value indicates more spamming activity. Let R and P be the entire set of reviewers and products. Let R(g) be the set of reviewers in a group g, and P(g) be the set of target products reviewed by the reviewers in R(g). Each reviewer i reviewed a set of products Pi.

To reduce the contingency of small groups, we use a penalty function L(g) which is a logistic function [Wang et al.2016]: L(g) = 1/(1 + e-(|R(g)|+|P(g)|-3) ). We subtract 3 from the sum since we consider minimum number of reviewers and products within a group to be 2 and 1, respectively; therefore L(g) ∈ [0.5, 1) [Wang et al.2018a].

Review tightness(RT). It is the ratio of the number of reviews in g (assuming each reviewer is allowed to write one review to each product) to the product of the size of reviewer set and product set in g.

If significant proportion of people co-reviewed several products, then it may imply a spam group activity.

Neighbor tightness(NT). It is defined as the average value of the Jaccard similarity (JS) of the product sets of each reviewer pair.

If the product sets are highly similar, both the reviewers are then highly likely to be together in a collusive group.

Product tightness(PT). It is the ratio of the number of common products to the number of products reviewed by all the members in g [Wang et al.2016].

Group members reviewing certain number of products and not reviewing any other products are more likely to indulge in fraud activities.

Rating variance(RV). Group spammers tend to give similar ratings while reviewing any product. Let S2 (p, g) be the variance of the rating scores of product p by reviewers in g. We take the average variance for all target products. The variance degree [0.5, 1) is converted into spam score between (0, 1].

Product reviewer ratio(RR). It is defined as the maximum value of the ratio of the number of reviewers in R(g) who reviewed product p ∈ P(g) to the number of all the reviewers of p, denoted by |Rev(p)|:

If a product is mainly reviewed by the reviewers in g, then the group is highly likely to be a spammer group.

Timewindow(TW). Fraudsters in a group are likely to post fake reviews during a short-time interval. Given a group g, and a product p ∈ Pg, we define the time-window based spamicity as

where SDp is the standard deviation of review time for a product p reviewed by reviewers in group g. T is a time threshold (set to 30 days in our experiments as suggested in [Wang et al.2018b]).

Algorithm 1 ExtractGroups

1: Initialize: 2: CCgroups ← set() ▷ Set of candidate groups 3: G(V,E) ←Edge attributed graph 4: CSet ← {} ▷ Potential merged groups 5: G,CCgroups ←GroupDetector(G,CCgroups) 6: Iterate: 7: G′ (V ′ ,E′ ) ←Attributed line graph of G 8: G′ ,CCgroups = GroupDetector(G′ ,CCgroups) 9: G←G′ 10: until |E| > 1 11: return CCgroups 12: function GROUPDETECTOR(G,CCgroups) 13: for each isolated node vi in G do 14: CCgroups.add(vi) and remove vi fromG 15: for each pair (ei, ej ) ∈ E × E do 16: if aei ⊂ aej then 17: if ⋂m∈aej Pm ⋃m∈aej Pm > 0.5 then 18: CSet(aei ) = CSet(aei ) ∪ aej 19: else 20: if ⋂m∈{aej \aei } Pm ⋃m∈{aej \aei } Pm > 0.5 then 21: CCgroups.add(aej \ aei ) 22: for each ei ∈ E do 23: k=CSet(aei ) 24: CCgroups.add(k) and remove k from G 25: for each connected component c with |c| > 2 do 26: CCgroups.add(c) and remove c from G 27: for each group g ∈ CCgroups do 28: if CollectiveScore(g) <= τspam then 29: CCgroups.remove(g) 30: return G, CCgroups 31: end function

3.2 Detection of Candidate Fraud Groups

We propose a novel graph-based candidate group detection method based on the "coherence principle".

Hypothesis 1 (Coherence Principle) Fraudsters in a group are coherent in terms of - (a) the products they review, (b) ratings they give to the products, and (c) the time of reviewing the products.

We show that each component of this hypothesis is statistically significant (see Sec. 5). We incorporate these three factors into a novel attributed product-product graph construction - G(V,E) such that each vij ∈ V indicates a product-rating pair (pi, rj) and its attribute avij consists of the set of reviewers who rated pi with rj. An edge e(ij,mn) = 〈uij , vmn〉 ∈ E indicates the co-reviewing and co-rating patterns of two products pi and pm with rating rj and rn, respectively. The edge attribute ae(ij,mn) indicates the set of coreviewers R(ij,mn) who reviewed both pi and pm and gave same ratings rj and rn respectively within the same time τt (defined in Sec. 3.3). Note that edge e(ij,in) connecting same product with different ratings wont exist in G as we assume that a reviewer is not allowed to give multiple reviews/ratings to a single product.

We then propose ExtractGroups (pseudecode in Algo 1, a toy example in Supplementary [Dhawan et al.2019]), a novel group detection algorithm that takes G as an input and executes a series of operations through GroupDetector() (Line 12) - isolated nodes are first removed (Lines 13-14), edge attributes are then merged and removed if Jaccard similarity (JS) of product sets that the corresponding reviewers reviewed is greater than a threshold (set as 0.5). Any group of reviewers eliminated due to the consideration of only common reviewers during the creation of edges is also checked through JS in order to avoid loosing any potential candidate groups (Lines 15-24). Before proceeding to the next iteration, connected components containing more than two nodes are removed (Lines 25-26). We define CollectiveScore as the average of six group level indicators defined in Sec. 3.1, and consider those as potential groups whose CollectiveScore exceeds τspam (Lines 27-29, defined in Sec. 5.2).

It then converts the remaining structure of G into an attributed line graph (edges converted into vertices and vice versa) G′ (V ′ ,E′ ) as follows: v′(ij,mn) ∈ V ′ corresponds to e(ij,mn) in G and av′ (ij,mn) = ae(ij,mn) ; an edge e′(ij,mn,ab) = 〈v′(ij,mn) , v′(ij,ab) 〉 represents co-reviewing and co-rating patterns of products pi, pm and pa; the corresponding edge attribute is a(ij,mn,ab) = ae(ij,mn) ∩ ae(ij,ab) . G′ is again fed into GroupDetector in the next iteration. Essentially, in each iteration, we keep clubbing reviewers together, who exhibit coherent reviewing patterns. The iteration continues until none of the edges remain in the resultant graph, and a set of candidate fraud groups are returned.

The worst case time complexity of ExtractGroups is O(k|E|2 ), where k is the number of iterations, and |E| is the number of edges in G (see Supplementary [Dhawan et al.2019] for more details).

Theorem 3.1 (Theorem of convergence)

ExtractGroups will converge within a finite number of iterations.

See Supplementary [Dhawan et al.2019] for the proof.

3.3 Ranking of Candidate Fraud Groups

Once candidate fraud groups are detected, DeFrauder ranks these groups based on their spamicity. It involves two steps - mapping reviewers into an embedding space based on their co-reviewing patterns, and ranking groups based on how close the constituent reviewers are in the embedding space.

Reviewer2Vec: Embedding Reviewers

Our proposed embedding method, Reviewer2Vec is motivated by [Wang et al.2018b]. Given two reviewers i and j co-reviewing a product p ∈ P by writing reviews cip and cjp with the rating rip and rjp at time tip and tjp respectively, we define the collusive spamicity of i, j w.r.t. p as:

where,

and γ control the importance of time, rating and the similarity of review content, respectively (0 ≤ α, β, γ ≤ 1 and α+β + γ = 1). We set γ > α, β as same review content signifies more collusion as compared to the coherence in ratings and time [Wang et al.2018b].

sp is the degree of suspicion of product p. τt, τr and θ are the parameters of the model. If the posting time difference among reviewers i and j on p is beyond τt or their rating on p deviates beyond τr (where τr = (max-min)x%, where max and min are the maximum and minimum ratings that a product can achieve respectively), we do not consider this coreviewing pattern. ExtractGroups achieves best results with τt = 20 and τr = (max-min)20% (see Sec. 5.3).

ζ(i, j, p) is the collusiveness between two reviewers w.r.t. product p they co-reviewed; however there might be other reviewers who reviewed p as well. Lesser the number of reviewers of p, more is the probability that i and j colluded. This factor is handled by sp after passing it through a sigmoid function. θ is a normalizing factor which ranges between [0, 1] (set as 0.4 [Wang et al.2018b]). We take the cosine similarly of two reviews cip and cjp after mapping them into embedding space using Word2Vec [Mikolov et al.2013].

Combining the collusive spamicity of a pair of reviewers across all the products they co-reviewed, we obtain the overall collusive spamicity between two reviewers:

We then create a reviewer-reviewer spammer graph [Wang et al.2018b] which is a bi-connected and weighted graph Gs = (Vs,Es), where Vs corresponds to the set of reviewers R, and two reviewers i and j are connected by an edge esij ∈ Es with the weight W s ij = Φ(i, j). Once Gs is created, we use state-of-the-art node embedding method to generate node (reviewer) embeddings (see Sec. 5.3).

Ranking Groups

For each detected group, we calculate the density of the group based on (Euclidean) distance of each reviewer in the group with the group's centroid in the embedding space. An average of all distances is taken as a measure of spamicity of the group. Let⃗i be the vector representation of reviewer i in the embedding space. The group spam score Spam(g) of group g is measured as:

4 Datasets

We collected four real-world datasets - YelpNYC: hotel/restaurant reviews of New York city [Rayana and Akoglu2015]; YelpZip: aggregation of reviews on restaurants/hotels from a number of areas with continuous zip codes starting from New York city [Rayana and Akoglu2015]; Amazon: reviews on musical instruments [He and McAuley2016], and Playstore: reviews of different applications available on Google Playstore. Fake reviews and spammers are already marked in both the Yelp datasets [Rayana and Akoglu2015]. For the remaining datasets, we employed three human experts 2 to label spammers based on the instructions mentioned in [Shojaee et al.2015,Mukherjee et al.2013a] 3 . The inter-rater agreement was 0.81 and 0.79 (Fleiss' multi-rater kappa) for Amazon and Playstore, respectively. Table 2 shows the statistics of the datasets.

5 Experimental Results

Statistical validation. To measure the statistical significance of each component (say, products reviewed by the group members) of Hypothesis 1 (Sec. 3.2), we randomly generate pairs of reviewers (irrespective of the groups they belong to) and measure how their co-reviewing patterns (cumulative distribution) are different from the case if a pair of reviewers co-occur together in the same group. We hypothesize that both these patterns are different (whereas null hypothesis rejects our hypothesis). We observe that the difference is statistically significant as p < 0.01 (KolmogorovSmirnov test). See Supplementary [Dhawan et al.2019] for more discussion. We then perform evaluation in two stages - quality of the detected groups, and quality of the ranking algorithms.

5.1 Baseline Methods

We consider three existing methods (see Sec. 2 for their details) as baselines for both the evaluations - (i) GSBP [Wang et al.2016],(ii) GGSpam [Wang et al.2018b],and (iii) GSRank [Mukherjee et al.2012].

5.2 Evaluating Candidate Groups

Along with the three baselines mentioned above, we also consider two variations of DeFrauder as baselines for group detection: DeFrauderR constructs the attributed productproduct graph G based only on co-rating without considering the time of reviewing; and DeFrauderT constructs G

2 They were social media experts, and their age ranged between 25-35.

3 https://www.marketwatch.com/story/

10-secrets-to-uncovering-which-online-reviews-are-fake-2018-09-21

based only on same co-reviewing time without considering co-rating. This also helps in justifying why both time and rating are important in constructing G for group detection. [Mukherjee et al.2012] suggested to use cumulative distribution (CDF) of group size (GS) and review content similarity (RCS) to evaluate the quality of the spam groups. Here we discard groups with size less than 2. Group Size (GS) favors large fraud groups as large groups are more damaging than smaller ones: GS(g) = 1 1+e-(|R(g)|-2) . Review Content Similarity (RCS) captures inter-reviewer review content similarity (as spammers copy reviews of each other): RCS(g) = maxp∈P(g){ 1 |R(g)|2 ∑(i,j)∈R(g)×R(g) cosine(cip , cjp )}.

The larger the deviation of each distribution from the vertical axis (measured in terms of Earth Mover's Distance (EMD)), the better the quality of the detected method [Mukherjee et al.2012].

Comparison. We choose the following parameter values as default based on our parameter selection strategy (Fig. 4): τt: 20 days, τspam = 0.4. The number of groups we obtain from different datasets is reported in Table 3. Fig. 2 shows the CDF of GS and RCS for all the competing methods on YelpNYC 4 , which is further summarized quantitatively (in terms of EMD) in Table 3. DeFrauder outperforms the best baseline by 11.35% and 4.67% higher relative EMD (averaged over four datasets) for GS and RCS, respectively. We also notice that DeFrauderT performs better than DeFrauderR, indicating that temporal coherence is more important than rating coherence in detecting potential groups.

5.3 Evaluating Ranking Algorithm

We use NDCG@k, a standard graded evaluation metric. Since each reviewer was labeled as fraud/genuine, we consider the graded relevance value (ground-truth relevance score used in Normalized Discounted Cumulative Gain (NDCG) [Manning et al.2010]) for each group as the fraction of reviewers in a group, who are marked as fraud. The candidate groups are then ranked by each competing method, and top k groups are judged based on NDCG.

Comparison. We choose the following parameter values as default based on our parameter selection strategy (Fig. 4):

4 Results are similar on the other datasets, and thus omitted.

α = β = 0.3, γ = 0.4, τt = 20 days, τr = (max-min)20% τspam = 0.4. We use Node2Vec [Grover and Leskovec2016] for embedding 5 . Fig. 3 shows the performance of the competing methods for different values of k (top k groups returned by each method). Since DeFrauder produces better groups (Sec. 5.2), we also check how the ranking method of each baseline performs on the groups detected of DeFrauder. Fig. 3 shows that with DeFrauder's group structure, GGSpam and GSBP show better performance than DeFrauder till k = 40; after that DeFrauder dominates others. However, all the baselines perform poor with their own detected groups. This result also indicates the efficiency of our group detection method. Table 3 reports that DeFrauder beats other methods across all the datasets except YelpZIP on which GSRank performs better with DeFrauder's detected groups. Interestingly, no single baseline turns out to be the best baseline across datasets. Nevertheless, DeFrauder outperforms the best baseline (varies across datasets) by 17.11% higher relative NDCG@50 (averaged over all the datasets).

5 We tried with DeepWalk [Perozzi et al.2014] and LINE [Tang et al.2015] and obtained worse results compared to Node2Vec.

Figure 4: Parameter selection on YelpNYC. We vary each parameter keeping others as default. Since τt = 20 produces best results for group detection, we kept this value for ranking as well.

6 Conclusion

In this paper, we studied the problem of fraud reviewer group detection in customer reviews. We established the principle of coherence among fraud reviewers in a group in terms of their co-reviewing patterns. This paper contributed in four directions: Datasets: We collected and annotated two new datasets which would be useful to the cybersecurity community; Characterization: We explored several group-level behavioral traits to model inter-personal collusive dynamics in a group; Method: We proposed DeFrauder, a novel method to detect and rank fraud reviewer groups; Evaluation: Exhaustive experiments were performed on four datasets to show the superiority of DeFrauder compared to five baselines.

Acknowledgements

The work was partially funded by Google Pvt. Ltd., DST (ECR/2017/00l691) and Ramanujan Fellowship. The authors also acknowledge the support of the Infosys Centre of AI, IIIT-Delhi, India.