SACNN: Self Attention-based Convolutional Neural Network for Fraudulent Behaviour Detection in Sports

1 Introduction

Sports events, such as the Olympic Games or FIFA World Cup, attract the attention of billions of people around the world. However, the fraudulent behaviour by athletes to enhance their performance in these events raises many social issues due to ethical and moral reasons [Rudenko, 2014]. The impact of this can be seen at both individual and societal levels, e.g., disqualification of athletes, or even ban of a nation from competing in future events, etc. [Kobierecka and Kobierecki, 2019] Therefore, it is a global concern that follows international sporting events worldwide, and antidoping analysis is a crucial measure to fight against these activities in sports [Callaway, 2011]. During the recent investigation at the Olympic Games 2014 in Sochi, a new form of fraudulent activity was found. Some athletes try to swap/exchange their doped samples with another individual's clean sample to evade positive tests, this refers to 'sample swapping' [McLaren, 2016]. This simple but new form of fraudulent activity became a threat to the whole anti-doping decision-making organisation. The anti-doping organisation maintains a longitudinal profile of every athlete, which contains records of all the samples collected from that athlete so far for the doping tests.

The primary way to detect sample swapping is to perform DNA analysis across all the samples [S´ıpoli Marques et al., 2005]. However, it is expensive and time-consuming to be implemented on the longitudinal profile of all the athletes who participated in the sports events (economic costs of more than $300 million/year). Other methods include monitoring each sample and comparing them with reference ranges of the athlete to be able to detect unnaturally high values [Rahman et al., 2022b; Piper et al., 2021; Sottas et al., 2006]. However, this interpretation does not take into account two important aspects. 1) Temporal behaviour: the human body or metabolism evolves over time, and so does the athlete's performance. Therefore, defining a reference range based on the previously collected samples would add bias when comparing the new samples. 2) Spatial behaviour: each collected sample is used to determine different parameters representing different elements of human steroid metabolism pathway, which have both intrinsic and extrinsic dependency [Schiffer et al., 2019; Rahman et al., 2023]. Therefore, it is important to exploit this information when comparing the similarity of the samples within the longitudinal profile.

This scenario can be well represented as a multivariate sequential anomaly detection problem [Ellore et al., 2020], where we are interested in determining the anomalous sequence based on spatio-temporal patterns. Many existing models that deals with sequential anomaly were studied [Lin et al., 2020; Zhao et al., 2021; Schlegl et al., 2017]. However, all these models mainly focus on manually defining the feature space, and fails to automatically learn the joint impact of spatial and temporal behaviour. Recently developed attention mechanisms have shown the benefit of automatic feature learning [Vaswani et al., 2017; Song et al., 2019]. The performance of a convolution network on spatio-temporal feature learning is also demonstrated in a wide range of tasks [Chen et al., 2021; Tran et al., 2015]. Therefore, in this paper, we present an approach that jointly considers spatio-temporal behaviour by constructing embedding maps and captures intrinsic relationships from these maps to uncover hidden fraud patterns. During experiments with real-world datasets, we show that the results of the proposed model significantly outperform other state-of-the-art (SoTA) baseline models. Our model serves as an adaptive approach for the preliminary screening. In case of positive results, confirmatory DNA testing is performed to eliminate false positives, ensuring that no athlete faces unjust penalties without irrefutable evidence.

The main contributions of our work can be summarised as follows:

• We propose a novel architecture based on self-attention mechanism, convolution layers and adversarial attack for detecting fraudulent behaviour in sports by capturing the embeddings from the longitudinal profiles of athletes. To the best of our knowledge, this is the first time that a fraud detection problem in sports has been addressed by considering spatio-temporal behaviour.
• Our method is extensively evaluated on different realworld datasets collected by anti-doping organisation and associated laboratory. The experimental results show the efficacy of our proposed model, which could detect more fraud athletes with relatively high specificity compared with SoTA baseline models.
• We perform a case study to show the performance of the proposed model on the real world fraudulent athlete's profiles which were tested by DNA analysis.

2 RelatedWork

2.1 Attentional Convolution Neural Network

Many recent studies have shown the advantage of combining an attention mechanism with convolutional networks for a wide range of applications [Vaswani et al., 2017; Song et al., 2019], such as medical image segmentation [Gu et al., 2021], language understanding [Shen et al., 2018], etc. For instance, the attentional convolution network is well exploited in many NLP-related tasks, e.g. text classification [Li et al., 2021; Xu and Cai, 2019], sequence-to-sequence prediction [Elbayad et al., 2018], document understanding [Nikolentzos et al., 2020], etc. [Cheng et al., 2020] employed an attention model for learning spatio-temporal features in fraud detection. Our approach develops from a similar intuition and integrates an attention network to generate embedding maps that consider both spatial and temporal aspects and let convolutional filters learn the relationships from these embedding maps.

2.2 Detection of Fraudulent Behaviour in Sports

Detection of fraudulent activities like doping using machine learning is not new in the sports anti-doping community.

[Sottas et al., 2006] proposed a Bayesian approach for the detection of abnormal values in the longitudinal profiles. Several studies [Van Renterghem et al., 2013; Wilkes et al., 2018; Wang et al., 2022; Rahman et al., 2022a] used different ML algorithms for detecting anomalous samples in the profile. However, the problem of investigating sample swapping is not so far addressed by machine learning. Currently, it is mainly detected by laboratory-based methods. Studies like [Thevis et al., 2012; Piper et al., 2021] showed how different biochemical techniques like gas chromatography-mass spectrometry, DNA-STR analysis, etc., can be used to detect sample swapping. However, these methods ignore the joint feature learning on spatial and temporal relationships. The approach we present in this paper is radically different and addresses this problem.

3 Preliminaries

3.1 Sample

A sample x = {f1, f2, ..., fk} means a urine sample collected from the athlete for performing the doping test, where each element represents the metabolites in human steroid metabolism. This metabolism pathway is a biological mechanism that follows spatial relationship [Rahman et al., 2023]. Testing sample xT refers to the sample under consideration for the similarity check with other samples in the longitudinal profile.

3.2 Longitudinal Profile

A longitudinal profile of an athlete X = {x1, x2, ..., xn} refers to a sequence of samples collected from that athlete at different times, where n = total number of samples. When n = 2, we define it as limited longitudinal profile Xlim = {x1, x2}. In this case, it is difficult to compute x1 ∼ x2.

3.3 Fraudulent Behaviour

A fraudulent behaviour in this paper refers to when an athlete performs sample swapping, i.e. exchange their doped sample with a clean sample from another individual. In this case, if the collected sample is xT , then it will not match with other samples in the longitudinal profile.

For example, let us assume we have the longitudinal profiles of athletes X1 and X2, where athlete X1 has a clean profile, i.e., all the samples are similar to each other and athlete X2 has anamolous profile. Let us suppose the samples x4, and x7 of athlete X2 are under consideration, i.e. xT = {x4, x7} whether they are from the athlete X2 or swapped by the athlete to evade the positive doping test. It could also be possible that both the samples are from the same athlete other than X2 (i.e. x4, x7 ∈ X1) or even from different athletes (i.e. x4 ∈ X1, x7 ∈ X3). Therefore, the goal is to find whether the given sequence is anomalous or not in the collection of sequences. For that, we need an iterative algorithm that investigates the similarity of each sample with every testing sample in the longitudinal profile, using xT ∼ xi, ∀ xT , xi ∈ X. However, the prevalence of sample swapping in the real-world situation is very less compared to the clean athletic population. Therefore, this problem can be well formulated as anomaly detection problem in multivariate sequential data.

4 Self Attention-based Convolutional Neural Network (SACNN)

Our proposed model consists of three main components: a subsequence generator, an attentional convolution neural network, and an aggregate function together with adversarial training, as shown in Figure 1.

4.1 Subsequence Generator

Input data is a collection of sequences I = {X1,X2, ...XN} with length of the sequences n1, n2, ...nN respectively. Since each sequence is of a different length, and the model requires an input of fixed dimensions, we generate subsequences of fixed dimensions out of the given sequence. The subsequence generator performs this operation in two steps. First it scans whether the sequence is a limited sequence Xlim. In this case, it is not possible to generate the subsequences, so the random generator randomly generates m additional samples based on sample x1 within the measurement uncertainty limit of ±10% and x2 can be treated as xT . It is a standard systematical uncertainty caused by the quantification instrument taken from biochemical domain experts [WADA, 2021]. The output of the random generator is the sequence consisting of x1, m generated samples, and x2. Next, the generator encodes each sequence into a set of subsequences denoted by E(X) where p represents the total number of generated subsequences for the given sequence X as shown below:

Each subsequence e has a fixed length denoted by le and consists of e = {xi, xi+1, ..., xle-1, xT}, xi, ..., xle-1 ∈ X. In this case, we compare the similarity of xT with the other elements in the subsequence. The generator generates sequences corresponding to all the possible combinations of the elements. This step is similar to the sliding window operation. However, the main difference is in this case, we want to consider all possible combinations of the elements of the sequence with xT into account and let the model learn the spatio-temporal relationship in all the combinations. The number of subsequences p can be calculated by:

where n represents the number of elements in the sequence X, and nxT represents the number of testing samples under consideration. These subsequences are then normalised separately. Therefore, the output is a set of normalised subsequences.

4.2 Attentional Convolution Neural Network

We propose a network architecture consisting of an input layer, four SAC units and a fully connected layer.

Input Layer

The input layer is Conv1D layer with 32 filters (1 × 1 × 32). It takes subsequence ei as input to perform a convolution operation and generates low-level embeddings for the given subsequence while preserving the spatial dimension. Therefore, the output is in a tensor format χ ∈ RN1×N2×N3 , where N1,N2,N3 denote length of subsequence, number of parameters, number of filters respectively.

SAC Unit

Each unit consists of self-attention layer, 2D convolution layer and batch normalisation.

1) Self-Attention Layer: The input tensor is first flattened using the reshaping layer. This is to make sure 2D embedding sequence is fed into the attention layer.

We use the self-attention layer for two reasons. Firstly, we are interested in mapping the spatio-temporal relationships of the embedding subsequence, i.e. compare each parameter of each element with itself. The attentional weights represent this relationship and can be used to generate high-level embeddings. Secondly, it increases the receptive field of the convolutional layer without adding computational costs associated with very large filter sizes.

The self-attention layer maps a query Qi and a set of keyvalue pairs (Ki, Vi) to an output. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed using the given query and the key. In our case, given the low-level embedding sequence from the input layer, the dot-product attention operation can be computed as:

The single attention layer performing h multi-head attention operation can be computed as:

where Qi = χ.wQ i , Ki = χ.wK i , Vi = χ.wV i and the learned attentional weights of the attention layer are:

The output of the attention layer consists of high-level embeddings with the same spatial dimensions, i.e. χatt ∈ RN1.N2×N3 . The reshaping layer changes back the dimension to χatt ∈ RN1×N2×N3 for the convolution layer.

2) 2D Convolution Layer: We use the convolution layer for three reasons. Firstly, it can learn the spatial relationship using filters, which is useful for this task. Secondly, it can learn more complex patterns from spatial space by stacking multiple filters. Lastly, the optimisation of the network could be efficiently performed by stochastic gradient descent algorithms on commercial hardware.

We use Conv2D layer with l filters (m×m×l) and padding to preserve the spatial dimension, where l = le. It can be represented by:

where C l i,j is lth filter which convolves over the embedding map χatt and represents the element-wise weights. Thus, the output feature map Sout i,j is obtained by different Conv2D filters:

where σ denotes the Leaky-ReLU activation function.

3) Batch Normalisation: Batch normalisation is used to reduce internal covariate shift caused by the attention and convolutional layers and for faster and more stable training.

where γ and β are learnable parameters and ˆχconv denotes normalised input to zero mean and unit variance.

Fully-Connected Layer

The output of the last SAC unit is first flattened using a global average pooling layer to χBN ∈ Rl . We use a fullyconnected layer which takes the flattened input and evaluates the probability of whether it has a fraudulent trade. If the probability is greater than the threshold value Pthres, the subsequence can be classified as anomalous.

where W denotes weight matrix and B represents bias vector. We used the cross entropy loss defined as:

where ˆ yi ∈ {0, 1} denotes the predicted label for the subsequence, and yi ∈ {0, 1} denotes the ground truth label, which is set to 1 if the subsequence is anomalous and 0 otherwise. f(χBN ) is the detection function that maps χBN to probability of whether the current subsequence is fraudulent. The proposed model can be optimised through the standard stochastic gradient descent algorithm. We used adam optimizer to learn the weights, and set the learning rate to 1e-3 and batch size to 256.

4.3 Aggregate Function

Since we get predictions ˆ yi for each subsequence separately from the fully connected layer. Therefore, we need an aggregate function to add each prediction and determine the final decision on the sequence. The aggregate function tells the likelihood of the given sequence being anomalous and can be defined as:

The final classification of the sequence can be calculated by:

where the value of Fthres is arbitrary and can be set accordingly to achieve high specificity.

4.4 Adversarial Training

We used adversarial training for two reasons. First, it adds novelty to the model by allowing it to recognise and classify variations in the input data that it may not have seen before. This can help to improve the model's generalisation capability and make it more robust to unseen profiles. Second, it is used as fairness-aware training that helps to eliminate discrimination or bias in the predictions. This can be done by incorporating fairness constraints into the training process, such as ensuring that the model's predictions are not systematically worse for certain demographic groups (e.g. based on gender, race, etc.). We employed SoTA adversarial attack called Generalised Universal Adversarial Perturbation (GUAP) [Zhang et al., 2020] to generate adversarial samples for the model. The goal of an GUAP attack is to generate single perturbations to the multiple input that cause the model to make a mistake. The generated samples can be represented as:

We used a loss function that penalises the model for producing outputs that are different from the original input. So the total loss function for the model can be represented as:

where ρ denotes the perturbation vector, α denotes perturbation constant, η represents learning rate, and λ controls the balance between the two loss functions.

5 Experiments

5.1 Datasets

We use real-world athlete datasets consisting of steroid longitudinal profiles with 11 parameters (Table 1) gathered by anti-doping agencies at various athletic events worldwide. The data is extracted from the Anti-Doping Administration & Management System (ADAMS) database [WADA, 2005] and [Rahman et al., 2022b] showed a detailed descriptive analysis of these datasets, including distributions, ranges, etc., where each dataset contains < 20% anomalous profiles, i.e., one sample is swapped in each profile (nxT = 1). In addition, the dataset Steroid-M and Steroid-F represent the case of limited longitudinal profile Xlim.

5.2 Baseline Methods

We employed the following SoTA models to compare the performance of our proposed model SACNN.

• Beta-VAE: [Higgins et al., 2017] Variational autoencoder uses modified reconstruction loss to find anomaly in a sequence.
• V-LSTM: [Lin et al., 2020] Sliding window based approach uses joint learning of VAE and LSTM to generate low-dimensional embeddings for anomaly detection.
• SUOD: [Zhao et al., 2021] Ensemble approach produce acceleration to different heterogeneous models for anomaly detection.
• XGBOD: [Zhao and Hryniewicki, 2018] Semisupervised boosting algorithm to extract useful embeddings from the sequence to detect outlier.
• LSCP: [Zhao et al., 2019] Unsupervised parallel ensemble algorithm which selects competent detectors in the local region of a sequential instance to detect outlier.
• AnoGAN: [Schlegl et al., 2017] Deep convolutional generative adversarial network that learns a manifold of normal anatomical variability to detect anomalies.
• IsoForest: [Liu et al., 2008] Unsupervised learning approach that constructs multiple trees which isolate observations with different characteristics to identify outliers.

5.3 Experimental Settings

We used the Steroid-All dataset [Rahman et al., 2022b] for training and validating all the models, which contains 50,450 clean profiles from both male and female athletes. In this dataset, we randomly selected 50% of profiles and manually swapped one sample in each profile with a sample from a different profile and labelled them as anomalous profiles (class 1). The other 50% of the profiles were labelled as clean profiles (class 0). Each profile is normalised separately, i.e. all the samples within the profile are normalised to the unit norm. We used 80% of the dataset for training all the models and 20% for the validation and evaluated the performance of SACNN against the baseline models. We need the model's performance at high specificity, which is a requirement from the anti-doping domain to avoid false negatives and unnecessary DNA testing (reducing unnecessary costs) and, hence, mimic the real-world situation. Therefore, we optimise the hyperparameters of all the baseline models to achieve optimal sensitivity under high specificity (99 ± 0.1)%.

6 Results

We compared the performance of SACNN with different baseline models on different datasets, as shown in Table 2. The uncertainties are evaluated using 5 fold cross-validation method. In all baselines, XGBOD and V-LSTM proven to be competitive, demonstrating the necessity of embedding extraction models for fraud detection. However, even with an accuracy of > 70%, Beta-VAE could not be able to detect any anomalous profiles (sensitivity of < 1%). In the case of limited profiles, we observe that all the models except SUOD show poor performance on Steroid-M dataset (in terms of sensitivity). However, for the Steroid-F dataset XGBOD and V-LSTM show better performance. The accuracy of all the models is much better because of the highly imbalanced nature of the datasets. Our proposed model outperforms all the baselines, i.e. generating spatio-temporal embeddings prove to be effective. SACNN achieves the sensitivity value of > 50% and AU value of > 80% on all the datasets.

6.1 Precision-Recall Curve

Figure 2 shows ROC and PRC curves for all the models evaluated on the Steroid-M dataset. As shown, the SACNN model performs better than all the baselines concerning both curves. The results of V-LSTM, SUOD and LSCP are quite similar.

All of them are much better than Beta-VAE. This might be because the fraud behaviour in longitudinal profiles is too complex for a simple autoencoder model to address. Among all the baselines, XGBOD is shown to be the most competitive. It might be because it generates a deep representation of parameters into embeddings using boosting algorithm.

Our proposed SACNN model consistently outperforms other SoTA baseline models. The reason is: (1) it deals with both spatial and temporal behaviour and generates embedding maps using an attention network, contrasted with XGBOD, which only deals with the spatial pattern and cannot address temporal behaviour of the longitudinal profile; (2) SACNN uses a convolutional network for better pattern learning from the generated embedding maps. Our model works even better at the very beginning of the curve compared to the other baselines. Moreover, our model can accurately detect more fraud longitudinal profiles with a high specificity.

6.2 Parameter Sensitivity

We studied the impact of different values of threshold parameters on the performance of our SACNN model. We varied both the threshold parameters (Pthres and Fthres) from 0 to 1 with a step of 0.1 and evaluated the sensitivity and specificity of the model. As shown in Figure 3, the sensitivity is dropping if we keep increasing the value of Pthres and Fthres, but it improves the specificity because of the tradeoff between the two. However, we observed that Pthres has a greater impact than Fthres because it is the threshold acting on the predictions of each subsequence individually. So, we set Pthres = 0.5 and Fthres = 0.5 as default setting and Pthres = 0.8 and Fthres = 0.6 to be high specificity setting i.e. 99% specificity value.

7 Case Study

To understand the spatio-temporal patterns of the longitudinal profile from the embedding maps, we performed a study on real-world proven cases. These longitudinal profiles were tested using DNA analysis performed by an accredited antidoping laboratory and found that 2 profiles were proven for sample swapping, 5 for doping and 22 for clean profiles. Our model could able to detect all the sample swapping and doping cases and 20/22 clean profiles. We selected one clean and one anomalous profile (sample swapping) and plotted the subsequence and the generated embedding maps from each SAC unit in Figure 4. The total number of embedding maps generated by the attention mechanism depends on the output of the Conv2D layer of the previous SAC unit. Therefore, these maps represent high-level embeddings. We plotted one of each embedding map to understand how the attentional weights in these maps are evolving. In the case of clean subsequence, we have higher weights in the embedding map of SAC unit 4 than anomalous subsequence, which shows a strong spatio-temporal relationship among three samples. Furthermore, the evolution of embedding maps also shows why we need at least 4 SAC units.

8 Ablation Studies

As shown in Table 3, we studied the effect of different components in SACNN. First, we removed the attention layer in the SAC units (w/o Att) and observed that the model's performance was degraded because the convolutional network is now learning the spatial relationship of the normalised subsequences instead of spatio-temporal embedding maps. This shows the importance of considering the spatio-temporal behaviour of the sequence. Second, we removed the adversarial attack in the model (w/o Adv) and observed that the model is less robust to the variation in input data. Next, we varied the number of SAC units in the model and observed that the model performs better when we keep adding a SAC unit until a point after which the performance starts dropping. The reason might be that adding a SAC unit helps to evolve the embedding maps, but once it is fully generated, adding more units will introduce overfitting. Moreover, adding SAC units exponentially increases the number of trainable parameters. Therefore, we selected 4 SAC units in the SACNN model.

To understand the significance of the subsequence genera- Table 3: Ablation studies showing the model performance evaluated on Steroid-M dataset at high specificity setting.

tor, we test two different variants of the model where instead of generating subsequence of length le, 1) we generate additional samples based on other samples to achieve the same length for all the sequences (w Add Samp). 2) we mask the additional samples with padding (w Mask). We observed that adding additional samples to shorter sequences will cause a bias in the model's decision since we know xT is not similar to masked or generated samples. Figure 5 shows the effect of different subsequence lengths le on the model's performance for both sliding window and our approach of considering all the combinations for defining the subsequences. We found 3 as an optimum subsequence length for our model.

9 Conclusion

In this paper, we propose an attention-based approach that takes into account the spatial and temporal behaviour of the longitudinal profile of athletes. It generates embedding maps using an attention mechanism and let the convolutional network capture the implicit relationship from these embedding maps. Our model achieves promising sensitivity at a high specificity value compared with SoTA baseline models. Furthermore, we explore the spatio-temporal patterns by observing the generated embedding maps in the case study on longitudinal profiles tested by DNA analysis. The results demonstrate that our model can effectively detect sample swapping and can help anti-doping authorities trigger fraudulent practices during sports events and make the sports clean.