Contrastive Graph Transformer Network for Personality Detection

1 Introduction

Personality refers to the characteristic pattern in a person's thinking, feeling, and decision-making [Kaushal and Patwardhan, 2018]. Personality detection is an emerging topic in user profile research, which aims to identify one's personality traits from online texts he/she creates and has expanded to massive applications such as recommendation system [Shen et al., 2020], dialogue system [Yang et al., 2021b; Wen et al., 2021] and computer game design [Lang et al., 2019].

†Equal contribution.

∗Corresponding authors.

With the blossoming of social media, users yield considerable posts containing their mental activities every day, offering new possibilities for automatically inferring personality traits [ˇStajner and Yenikent, 2020]. Earlier researchers mainly used two sources of lexical features, Linguistic Inquiry and Word Count (LIWC) [Tausczik and Pennebaker, 2010] and Medical Research Council (MRC) [Coltheart, 1981] to identify personality from user-generated posts [Mairesse et al., 2007]. To overcome manual feature engineering, deep neural networks (DNNs) were applied in the personality detection task to obtain the representations of posts. However, understanding the hidden personality traits behind the posts is non-trivial. Most recent works have been devoted to refining post representations from the perspective of the post structure including [Lynn et al., 2020], [Yang et al., 2021c],and [Yang et al., 2021a]. Despite the considerable improvements achieved in personality detection, the existing models are likely to suffer from the scarcity of personality tags as the ground-truth personality traits are usually collected from professional questionnaires, which are often resource-intensive and time-consuming. Hence, such precious personality tags are hard to collect, which becomes a limitation for training deep neural networks and makes it difficult to infer personality from posts.

In addition, personality is defined in terms of different dimensions (traits) and these traits often co-occur with a nonnegligible correlation, which has been confirmed in empirical psychological researches[John et al., 2008; Sharpe et al., 2011]. For example, neurotic people are more likely to be extroverted, like Trump. However, such implicit trait correlations are rarely exploited, which should have been the key psychological cues to be considered for personality detection.

Taking both the problem of data scarcity and the correlations among traits into consideration, we model the usergenerated posts as a fully-connected post graph, and propose a novel Contrastive Graph Transformer Network model (CGTN) for personality detection, which distills potential labels of the graphs based on both labeled and unlabeled data. Specifically, CGTN consists of a contrastive post graph encoder and a trait sequence decoder. In post graph encoder, two types of graph augmentations are designed to incorporate different priors based on psycholinguistic knowledge of LIWC and post semantics. To be precise, LIWC can be utilized to extract psycholinguistic features while post semantics are able to capture the semantic relations among the posts. A self-supervised paradigm is defined to maximize the agreement over the representations of the augmented graphs that come from the same user. This contrastive strategy allows us to learn the post embeddings without using any labeled data. In trait sequence decoder, we view the multi-trait detection task as a trait sequence generation problem and apply a transformer-based decoder to model the correlations among traits. In addition, we use the post-to-trait attention to ensure that crucial posts are selected for trait generation.

In summary, our main contributions are as follows:

• To our best knowledge, this is the first effort to explore contrastive self-supervised learning to distill auxiliary signals for personality detection, providing a new perspective for alleviating the data scarcity for personality detection.
• We propose a novel Contrasive Graph Transformer Network (CGTN) model, for which we design two types of graph augmentations to incorporate priors based on LIWC, and post semantic knowledge, and explicitly introduce trait correlations by exploiting a sequence generation model.
• The experimental results demonstrate the outperformance of our model over the baselines including the state-of-the-art methods, which shows the effectiveness of our model.

2 RelatedWork

As an emerging interdisciplinary study, personality detection has attracted the attention of both computer scientists and psychologists [Xue et al., 2018; Mehta et al., 2020; Yang et al., 2021c].

Earlier studies mainly exploit psychologically statistical features to detect personality [Mairesse et al., 2007], such as LIWC [Tausczik and Pennebaker, 2010] and MRC [Coltheart, 1981]. Nonetheless, the statistical analysis cannot effectively represent the original semantics of the posts. With the rapid development of deep learning, a series of Deep Neural Networks (DNNs) are applied to personality detection task and have achieved great success, including CNN [Xue et al., 2018], LSTM [Tandera et al., 2017], etc. Recently, personality detection has benefited from large-scale pre-trained language models, such as BERT [Devlin et al., 2018], and thus get improved[Mehta et al., 2020; Ren et al., 2021]. Based on these pre-trained models, latest works focus on refining post representations from the perspective of post structure. [Lynn et al., 2020] designed SN+Attn which introduces a hierarchical attention network to obtain user document representations in a bottom-up manner from the word-grained level to the post-grained level, arguing that not every post contributes equally. In order to avoid introducing post-order bias, Transformer-MD [Yang et al., 2021a] considers different posts to be unrelated to each other. TrigNet [Yang et al., 2021c], however, holds a different view that there is a psycholinguistic structure between posts and constructs a heterogeneous graph for each user and aggregates post information from a psychological perspective.

However, the above methods mainly focus on obtaining the representations of the user's posts by the supervised paradigms. For personality detection task, human-provided labels are hard to collect, thus the model tends to overfit the training data and performs poorly on test data. In this work, to address the issue, we develop a novel contrastive graph transformer model for personality detection, which fully exploits both labeled and unlabeled data through contrastive self-supervised learning.

3 Preliminaries

Personality detection can be phrased as a multi-document multi-label classification task [Lynn et al., 2020; Yang et al., 2021a]. Formally, given a set P = {p1, p2...pn} of N posts from a user, where pi = {w1 i ,w2 i ...wk i } is i-th post with k tokens, our goal is to predict t-dimensional personality traits from the trait-specific label space Y ={y1, y2, · · ·yt}, e.g., t = 4 in the MBTI taxonomy, t = 5 in the Big-Five taxonomy. In this paper, we model a user-generated document as graph over posts. For each user with n posts, we construct a fullyconnected original graph G = (V, E), where V consists of n post nodes and the edges E capture the correlations among the posts. The BERT is employed to obtain the initial embeddings of each post node. And then based on the post graph, we propose a Contrastive Graph Transformer Network model (CGTN) for personality detection.

4 Contrastive Graph Transformer Network

Figure 1 presents the overall architecture of the proposed CGTN, which consists of a contrastive post graph encoder and a trait sequence decoder. The encoder aims to learn rich post representations via self-discrimination on post graph while the decoder is to uncover psychological cues contained in the personality correlations. In the following subsections, we detail the contrastive post graph encoder and trait sequence decoder.

4.1 Contrastive Post Graph Encoder

In contrastive post graph encoder, we design two types of graph augmentations based on psycholinguistic knowledge of LIWC and post semantics. Thereafter, contrastive selfsupervised learning is exploited on augmentations graph to learn post representation by judging whether two augmented graphs are from the same user.

Post Graph Augmentation

The core of personality detection is to understand a collection of user-generated posts. Previous works demonstrated that digging the inherent patterns in the structure of post is helpful for representation. Self-supervised learning allows us to exploit the "unlabeled" data via making disturbs on the input data. Naturally, we can construct the "unlabeled" data by generating multi-view post graphs for each user. Specifically, LIWC is used to construct psycholinguistic view graphs Gα [Yang et al., 2021c]. The LIWC dictionary divides words into psychology-related categories C = {c1, c2 · ··, cn} which can be taken as bridges to connect different post nodes. Two post nodes are connected if they contain the words of the same categories. For the semantic view graphs Gβ , We build the edges between the posts if their semantic similarity is larger than a given threshold. The semantic similarity is computed as the cosine similarity based on the initial post embeddings.

Graph Contrastive Self-supervised Learning

Contrastive self-supervised learning offers a simple way to learn invariant representations by local disturbs in the input data without using any labeled data. In our task, we randomly sample a batch of U users and any pair of augmented graphs (Gα,Gβ ) that comes from the same user is considered as a positive pair. Otherwise, they are labeled as negative. We learn to predict whether two augmented graphs originate from the same user or not. In the following, we first introduce how we learn the representation of a graph and then illustrate the contrastive loss.

Specifically, to obtain the graph representation, we first use GNN to capture the structural information within nodes' neighborhoods [Xu et al., 2018]. The L-th layer GNN updates the post node embeddings hp as:

where p′ is the neighbour node of post node p on the given augmented graph. where hLp is the embedding of the node p at the L layer. After obtaining the post node embeddings with fused neighbor information, we pass them through an average pooling layer and a two-layer MLP to obtain the entire graph representation. Formlly,

Based on the above graph embedding, the psycholinguistic augmentation graph and the semantic augmentation graph of user u are represented as zα u and zβ u , respectively. Given a positive pair (zα u , zβ u ) and a negative pair (zα u , zβ v ), which is sampled from the augmented graphs of other users v within the same batch. The contrastive loss Lcl is defined to maximize the consistency between positive pairs compared with negative pairs:

where sim() denotes the cosine similarity and τ is a temperature hyperparameter.

4.2 Trait Sequence Decoder

Unlike single-trait classification where only one label is assigned to each sample, a decoder with Transformer [Vaswani et al., 2017] backbones is designed to capture the correlations of traits by the sequence generation architecture. In addition we design post-to-trait attention to select the key posts for trait generation. Formally, the trait generation can be modeled as finding an optimal trait sequence y∗ that maximizes the conditional probability:

where Hα u = [hαp1 , hαp2 , · · ·, hαpn ] is post sequence based on psycholinguistic view graph Gα, similarly, Hβ u is post sequence based on semantic view graph Gβ .

The decoder as shown in the right part of Figure 1 is composed of M identical blocks, where each block contains a multi-head self-attention layer, a post-to-trait attention layer and a feed-forward layer. Formally, the output of the first sublayer Cm, the second sub-layer Dm, and the third sub-layer Em at m-th decoding block are sequentially calculated as:

where LN( · ) denotes layer normalization, SATT( · ) denotes multi-head self-attention mechanism, PTATT( · ) is post-totrait attention layer we inserted, and FFN( · ) is feed-forward network, Hu = {Hαu ,Hβu } denotes two post sequences, respectively.

Post-to-trait Attention

We design post-to-trait attention sub-layer to select crucial posts from the two augmented views for generating traits. This inserted sub-layer includes two steps: first, two viewspecific post sequences Hαu and Hβu are fed into the decoding module simultaneously. For each decoding step, the decoder processes each view independently and obtains two contextual sequences (Cmp→t )α and (Cmp→t )β . Formally:

Subsequently, we leverage cross-view self-attention over two sequences to control different contributions of different views at each step. Formally:

Trait Generation

Finally, the output of the last layer of the decoder Em is used to detect the personality via linear and softmax layer. The generation of the t-th trait by the decoder can be formalized as

where It is the mask vector at decoding step t that is used to prevent the decoder from detecting the repeated trait. In inferring stage, ̂ yt is further used as input token of the next generation step to detect the (t + 1)-th trait:

(11)

Following SGM [Yang et al., 2018], we use beam search to find the top-ranked prediction path at generation time. The final output is trained using the mean binary cross-entropy over all traits. Given true binary label vector yt and predicted labels ̂ yt, the detection loss is:

4.3 Model Training

We apply two training strategies including pre-training and joint learning. For the pre-training strategy, the model is trained in a two-stage paradigm. Given a collection of unlabeled post graphs, a direct contrastive method is to predict whether two augmented graphs are similar. After training, we finetune the pre-trained graph embeddings in the downstream trait generation task. For the joint learning strategy, an auxiliary self-supervised task is included to help learn the supervised detection task, and two tasks share the same graph encoder. Our training objective is to minimize the cross-entropy loss and contrastive loss corresponding to the tasks of personality detection and post graph contrastive self-supervised learning, respectively. Formally, the objective function is defined as follows:

where λ is a trade-off parameter to control the strengths of contrastive learning Lcl.

5 Experiments

5.1 Dataset

Following previous studies, we conduct experiments on the Kaggle 1 with MBTI taxnomy and Essays datasets with BigFive taxonomy. The Kaggle dataset is collected from PersonalityCafe, where people share their personality types and daily communications, with a total of 8675 users and 45-50 posts for each user. The traits for Kaggle dataset, namely, MBTI taxonomy, include Introversion / Extroversion, Sensing / Ntuition, Think / Feeling, and Perception / Judging. The Essays [Pennebaker and King, 1999] is a well-known dataset of stream-of-consciousness texts which contains 2468 anonymous users with approximately 50 sentences recorded for each user. Each user is tagged with a binary label of the Big Five taxonomy, including Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism. Two datasets are randomly divided into 6:2:2 for training, validation, and testing, respectively. The F1 metric is adopted to evaluate in the Essays dataset. The Macro-F1 is adopted to evaluate the performance in each personality trait since the Kaggle dataset is imbalanced. Note that, due to the privacy and high expenses for data collection, available personality datasets with standard labels are rare. In 2018, the MyPersonality dataset 2 stopped sharing as the world's largest personality dataset due to privacy breach.

5.2 Baselines

We compare our model with several baselines, which can be categorized as follows.

• BiLSTM [Tandera et al., 2017] is a sequence model firstly employed to encode each post, and then the averaged post representation is used for user representation.
• AttRCNN [Xue et al., 2018] is a hierarchical structure, in which CNN-based aggregator is employed to obtain the user representations.
• BERT is a pre-trained language model, [Mehta et al., 2020; Ren et al., 2021] perform extensive experiments to arrive at the optimal configuration for personality detection.
• SN+Attn [Lynn et al., 2020] is a hierarchical network, in which the GRU with attention is used to encode both sequences of words and posts for user representations.

1 kaggle.com/datasnaek/mbti-type

2 http://mypersonality.org./

• TrigNet [Yang et al., 2021c] is a novel flow tripartite graph attention network, which aggregates different posts of each user from a psychological perspective.
• Transformer-MD [Yang et al., 2021a] is a novel multidocument Transformer, which aggregates different posts to depict a personality profile for each user without introducing post orders.

5.3 Implementation Details

Following previous works [Yang et al., 2021c; Yang et al., 2021a], we set the max number of posts as 50 for each user and the max length as 70 for each post. For pretraining, the initial learning rate is searched in { 1e-2 , 1e-3 , 1e-4 } and to optimize the contrastive loss on different datasets. The mini-batch size is set as 64. The temperature τ is set as 0.15. We adopt early stopping when the validation loss stops decreasing by 10 epochs. For joint learning, we search the trade-off parameter λ in { 1, 0.1, 0.01, 0.001, 0.0001} for different datasets. The initial learning rate is also searched in { 1e-2 , 1e-3 , 1e-4 }. The settings of batch size, patience for early stopping, and temperature are the same as the pretraining strategy 3 .

5.4 Overall Results

The overall results are presented in Table 1. The major findings can be summarized as follows, First, we can observe that our final model CGTNjoint achieves the highest scores for both datasets, significantly outperforming the current state-of-theart model (TrigNet) by 2.75 with t-test p<0.01 in Kaggle dataset and 4.98 with t-test p<0.01 in Essays dataset. What's more, with pre-training strategy, our model CGTNpretrain also achieves significant breakthrough compared to the current SOTA model TrigNet. The results verify the effectiveness of our model in personality detection. We believe the reasons are two fold: (1) Our model CGTN uses contrastive self-supervised learning to learn better post representations which reduces the risk of overfitting on a small training set. (2) Trait correlations are well captured, which injected some psychological clues into for personality detection. Second, CGTNpretrain and CGTNjoint performs better on Essays dataset compared with baseline, we measure the difference between the corresponding scores of the training and testing sets compared to that of the supervised paradigm approach TrigNet on each datasets and find that the train-test difference under CGTNpretrain, joint is smaller than that under TrigNet. This is even more obvious on Essay dataset which indicates that our method can better mitigate overfitting under small datasets. Third, CGTNjoint generally performs better than CGTNpretrain. As shown in Table 1, the train-test difference under CGTNjoint is smaller than CGTNpretrain on two datasets, which indicates that the fine-tuned representations are still at risk of bias towards overfitting. Joint learning strategy is probably the better option since the representations in the main and auxiliary tasks are mutually enhanced with each other. Fourth, TrigNet and Transformer-MD achieve greater performance compared to the DNNs models, which further implies that making full use of post structure information is essential for personality understanding.

3 The code available at https://github.com/yangpu06/CGTN

5.5 Ablation Study

We conduct an ablation study of our CGTNjoint model on both datasets by removing trait correlation component, represented by CGTNw/o TC, and contrasive learning component, represented by CGTNw/o CL, to investigate their contributions respectively. As shown in Table 2, we observe that CGTNjoint outperforms CGTNw/o TC, suggesting the effectiveness of our approach in modeling trait correlations. In particular, the performance improvement on the Essays dataset is higher than on the Kaggle. We guess that it might be the correlations of Big Five personality traits are slightly higher than that of MBTI indicators. In addition, the performance of CGTNjoint is also supreior to that of CGTNw/o CL, especially on Essays dataset, which shows contrasive learning is helpful for increasing the generalization ability of the model, especially on small datasets.

5.6 Impact of Number of Training Samples

We compare our model with two baseline methods with the best performances: Transformer-MD and TrigNet, to study the impact of the ratio of training set. Particularly, we vary the number of training samples and compare their performance on the Kaggle and Essays dataset. We run each method 10 times and report the average performance. As shown in Figure 2, with the increase of training data, all the methods achieve better results in terms of Macro-F1 and F1 on both datasets. Generally, our method outperforms all the other methods consistently. When fewer training data are provided, the baselines exhibit obvious performance drop, while our model still achieves relatively high performance. It demonstrates that our method can more effectively take advantage of the limited labeled data for personality detection. We believe our model benefits from auxiliary signals distilled through contrastive self-supervised learning for personality detection.

5.7 Effect of Trade-off Parameter

Figure 3 demonstrates how Macro-F1 and F1 values change when the trade-off parameter λ in CGTNjoint increases. from which we can observe that the score first rises as the tradeoff parameter λ rises and then begins to drop when λ is larger than 0.1. This is because a bigger value imposes a stronger regularization impact, which helps to reduce overfitting. However, if λ gets too high, the score will drop because excessive regularization impact outweighs the detection loss.

5.8 Training Efficiency

We investigate the effect of self-supervised contrastive learning on training efficiency. Figure 4 shows the training curves of CGTNjoint and CGTNw/o CL on Kaggle and Essays datasets. Obviously, CGTNjoint converges much faster than CGTNw/o CL on both datasets. In particular, early stop occurs at the 35-th epochs and arrives at the best performance for CGTNjoint, while it takes more epochs for CGTNw/o CL on Kaggle dataset. It demonstrates that contrastive learning task speeds up the detection progress and helps to learn a better model. The Essays dataset shows the same trend, and CGTNjoint has a lower training loss. The above results verify that the proposed contrasive self-supervised paradigm is effective for such a data-hungry task.

6 Conclusion

In this paper, we proposed a novel Contrastive Graph Transformer Network model (CGTN) for personality detection. CGTN aims to introduce a new learning paradigm to alleviate the data scarcity inherent to personality detection tasks. For this purpose, we designed two types of graph augmentations based on LIWC and post semantics and learned post embeddings from graph self-supervised contrasive learning. Besides, Transformer-based trait generation architecture is designed to exploit correlations among personality traits. Moreover, we used post-to-trait attention to select the vital posts for trait generation. In the end, extensive experimental results on Kaggle and Essays datasets demonstrate the effectiveness and efficiency of our model.

Acknowledgments

This work is supported by the NSFC-General Technology Basic Research Joint Funds under Grant (U1936220), the National Natural Science Foundation of China under Grant (61972047) and the National Key Research and Development Program of China (2018YFC0831500).