Dual-View Self-Supervised Pre-Training for Expert Finding

1 Introduction

Expert finding methods route questions to suitable answerers on Community Question Answering (CQA) websites, and have attracted considerable attention in recent years [Hu et al., 2019; Hu et al., 2021; Li et al., 2022; Zhao et al., 2017; Qian et al., 2022; Qian et al., 2023]. The task requires learning well representations of both experts and questions from historical interactions, so that an incoming question can be matched to a capable expert.

∗Corresponding author

Existing methods vary in how they balance the modeling of experts and questions. A first line prioritizes question modeling. Doc2Vec-based and early neural methods [?; Shahriari et al., 2015] produce rich question representations, yet represent experts through simple aggregation (e.g., mean or max pooling) over their answered questions, which collapses diverse answering preferences into coarse vectors. A second line of work prioritizes expert modeling. ExpertPLM [Peng and Liu, 2022] and ExpertBert [Liu et al., 2022b] design pre-training tasks tailored to expert behaviors and achieve fine-grained expert representations, but treat questions as information inputs and encode them with relatively shallow strategies, leaving question semantics underexploited. A third line attempts to give equal attention to both sides. NeRank [Li et al., 2019], RMRN [Fu et al., 2020] and PMEF [Peng et al., 2022] learn expert and question representations jointly and obtain strong performance, yet they rely on supervised expert-question interactions for representation learning, and such interactions are extremely sparse on CQA platforms. For example, the interaction densities on the Es and Gis datasets from StackExchange 1 are only 0.0429% and 0.0435%, which fundamentally limits the quality of learned representations regardless of modeling strategy. In short, existing work suffers from two intertwined limitations: an imbalanced modeling granularity across the expert and question views, and a universal dependence on sparse supervised signals.

Self-supervised pre-training remarkably successful at distilling general representations from unlabeled data in NLP [Devlin et al., 2018; Peng et al., 2025] points to a natural direction. However, adapting it to expert finding is non-trivial. Standard PLMs operate at the sentence and token level, whereas an expert is not a sentence but a sequence of historical answering behaviors, and token-level objectives such as Masked Language Modeling cannot distinguish the inherent preference differences between experts. Besides, expert representations are learned only indirectly, as a byproduct of fitting expert-question matching labels. There is no explicit mechanism that pushes the model to distinguish one expert's personalized preference from another's, so experts with overlapping answer vocabularies but different underlying interests tend to collapse into similar representations. This leaves two specific questions: (i) How to design a dual-view pre-training architecture that handles expert and question representations simultaneously? and (ii) How to design view-specific self-supervised tasks that capture the unique characteristics of each view?

1 https://archive.org/details/stackexchange

To this end, we propose a Dual-view Self-Supervised pretraining framework for Expert Finding (SSEF). The core is that we design two view-specific pre-training tasks for expert and question representation learning respectively from large unlabeled data to address the two aforementioned questions.

For question-i, we construct a dual-view pre-training corpus including: (1) question view, which contains the title of each question (one question, one input line) from CQA websites; (2) expert view, which aggregates the historical questions answered by one expert as one sequence (one expert, one input line). We further design two special tokens ([ER] and [QR]) at the beginning of two sequences to help the PLMs identify whether the input is an expert or a question. In this way, the model is endowed with the ability to capture expert and question representations simultaneously. For question-ii, one of the crucial modules in self-supervised pretraining is data augmentation. Hence, considering that the expert interest could be learned from the historical behaviors (i.e., the answered questions), we propose two expert-view data augmentation methods (namely behavior crop and reorder) to augment the original expert-view information. Then the model can be optimized with expert-view contrastive learning. In this way, our method can pre-train expert representations by distinguishing the preference difference between experts. For questions, we employ two data augmentation strategies on the question-view information (namely word crop and reorder) and pre-train question features by distinguishing the intrinsic difference between questions.

The expert-/question-view self-supervised modules are unified by multi-task learning and share the parameters. In the fine-tuning phase, we utilize the pre-trained weight to encode expert/question from their respective views, and then accomplish the downstream expert finding task. In summary, the contributions of our method are:

1. We design a self-supervised pre-training framework for expert finding, and this pre-training and fine-tuning paradigm could alleviate data sparsity.
2. The model pre-trains experts and questions simultaneously. Besides We design new data-augmentation approaches for expert-view and question-view self-supervised learning.
3. To verify the effectiveness of SSEF, we perform extensive experiments on six real-world CQA datasets. Experimental results demonstrate that our method can achieve better performance than existing baselines and validate the effectiveness of our approach.

(4) In zero-shot setting, our method still shows competitive results. Another observation is that our pre-training weight could further improve other methods' performance.

2 RelatedWorks

2.1 Expert Finding

To find capable experts for providing satisfactory answers to questions on CQA websites, the expert finding has received much attention from both research and industry community [Zhu et al., 2011; Liu et al., 2015; Roy et al., 2018; Yuan et al., 2020b; Wang et al., 2024; Amendola et al., 2024; Etemadi et al., 2024]. Previous studies about expert finding could be categorized into traditional methods and deep learning-based methods. In most traditional methods, feature-engineering [Cao et al., 2012] or topic-modeling technology [Guo et al., 2008; Riahi et al., 2012; Wei et al., 2016] is employed to model the questions and experts, then routed target questions to the suitable experts. Deep learningbased methods employed the neural network to model experts and questions, then matched experts and target questions [Li et al., 2019; Fu et al., 2020; Ghasemi et al., 2021; Liu et al., 2022b]. For example, TCQR [Zhang et al., 2020] employed a temporal context-aware attention model to learn the answerer representation, which could model temporal dynamics by multi-resolution settings.

Different from existing methods of learning expert and question representations that usually rely on supervised data, we design a self-supervised pre-training paradigm, which could integrate self-supervised signals from large unlabeled data and pre-train expert and question representations.

2.2 Self-Supervised Learning

Self-Supervised Learning (SSL) [Liu et al., 2020; Gui et al., 2024; Ren et al., 2025] have shown outstanding performance on various machine learning tasks, especially in Computer Vision [He et al., 2020; Newell and Deng, 2020], Natural Language Processing [Devlin et al., 2018], etc. For example, BERT [Devlin et al., 2018] used masked language models to capture deep bidirectional representations, which has been utilized in various domains [Xu et al., 2023]. However, 1) they almost are designed on corpus-level or sentence-level for learning general language knowledge; 2) data augmentation tasks in these methods (e.g., token shuffle/masking) are almost token-level, which would be not suitable for modeling expert. Hence, designing appropriate data augmentation tasks for expert finding is necessary.

It is noted that ExpertPLM [Peng and Liu, 2022] and ExpertBert [Liu et al., 2022b] are also expert finding models with self-supervised paradigms. However, they 1) only focus on expert modeling and omit the question modeling simultaneously; 2) usually mask words in the expert-view input information for expert interest modeling, which is not customized for expert finding and could not distinguish the inherent interest difference between experts.

3 Problem Statement

In this section, we formalize the expert finding. We represent column vectors and matrices by bold lower case letters (e.g., e) and bold upper case letters (e.g., X), respectively. Then, we formulate the problem definition of expert finding in our paper. Suppose that there is a target question qt and a candidate expert set C = {c1, · · · , cN} respectively, where N is the number of experts. We need to compute the matching score sc measuring the relevance between the expert in C and the target question qt . Then, candidate experts are ranked and recommended to the target question qt based on the sc.

Furthermore, the candidate expert ci ∈ C is associated with a set of her/his historical answered questions, which can be denoted as Ei = {q1, · · · , qm}, where m is the number of historical questions. Any question is represented by its related title, which is denoted as a word sequence Q. Note that the expert who provides the "accepted answer" to the question will be regarded as the ground truth.

4 ProposedMethod

In this section, we will introduce the training procedure of our dual-view method SSEF, which is composed of the following stages:

(1) In pre-training, the expert-/question-view pre-training inputs are first well constructed respectively. Then we propose expert-/question-view data augmentation methods to enhance original expert/question information and perform selfsupervised tasks to fully model expert/question from unlabeled input data respectively. (2) In fine-tuning, we initialize the parameters of the expert or question representation model with the pre-trained parameters and then utilize traditional expert finding supervised signals to fine-tune the model.

4.1 Dual-View Pre-Training Framework

In this part, we will present the expert/question pre-training framework, including the well-designed input layer and a brief backbone architecture introduction.

For expert-view input, we concatenate the words of the expert's historically answered questions into a whole sequence as shown in Figure 1. We add [HSEP] tokens between histories to distinguish different historical questions, and the special token [SEP] at the end of the input word sequence for recognizing the end of the sequence. Moreover, we propose to design a special token ([ER]) at the beginning of the input sequence, for indicating the expert information and help obtaining representations. Hence, the expert-view information is denoted as:

Likewise, we propose a special token [QR] at the beginning of the question-view information to facilitate obtaining the

̅

question representation, which is defined as follows:

The special tokens [ER] and [QR] are not just for capturing the representation of experts and questions. More importantly, they could help the model identify whether the input data is an expert or a question, which can be employed throughout the pre-training and fine-tuning.

Then, the input representation matrix X is constructed by summing its corresponding token, segment, and position embedding:

It is noted that we don't employ the Next Sentence Prediction task during pre-training, and hence, we set all the segment IDs to 0.

In our method, we adopt the widely used BERT [Devlin et al., 2018] as our base backbone for further pre-training.

4.2 Data Augmentation

In this part, we will show the well-designed data augmentation methods for experts and questions before pre-training.

1. Expert-view Augmentation For generating selfsupervised signals to pre-train expert representation from historical questions, we design two augmentation approaches (shown in Figure 2 (a)(b)), which could construct different views of the same sequence but still maintain the main preference hidden in historical interactions.

(1) Behavior Reorder In fact, we could obtain satisfied recommendation lists without strictly following the chronological order [Yuan et al., 2019; Yuan et al., 2020a]. Similarly, on CQA websites, it is likely that the expert answer questions in any order. For example, the sequence of historical questions is [q1-"Single word for plaintiff or defendant", q2-"Geometric or Geometrical?", q3-"Replacement for this means that ..."]. This is almost equivalent to [q2, q1, q3], since there is no strict order between questions.

Hence, we can derive a self-supervised augmentation operator with the assumption of flexible order. Specifically, given an expert historical question sequence Ei = {q1, · · · , qm}, we randomly shuffle a continuous sub-sequence with augmentation ratio αb (the length is αb ∗ |Ei|), denoted as:

In this way, the SSEF could model the expert by relying less on the order of interaction sequences.

(2) Behavior Crop In fact, randomly cropping a continuous sub-sequence in E might not affect the expert modeling greatly. This augmentation is designed to identify the influential historical questions for modeling expert, and make that less sensitive to the complete historical sequence. More concretely, given an expert historical question sequence Ei = {q1, · · · , qm}, we randomly crop a continuous sub-sequence (with the length βb ∗ |Ei|), which is denoted as:

In this way, the SSEF could capture a local view of the expert histories and model more generalized expert representation without comprehensive information of experts.

4.2.2 Question-view Augmentation. We employ two augmentation approaches on the question-view information to construct different views of that. As shown in Figure 2 (c)(d), given the i-th question word sequence Qi, we employ word reorder [Yan et al., 2021] or word crop augmentation strategies to change a continuous sub-sequence in one question, and the augmentation ratio is αw or βw (e.g., the changing length is αw ∗ |Qi|). The operators are defined as follows:

It is noted that we only shuffle position IDs while keeping the order of the token IDs in the word reorder. This procedure encourages the model to identify the unique information that is shared across different views of the same question input, which helps learn more generalized question embeddings.

4.3 Self-Supervised Pre-Training Task

Having established the augmented views of the expert or question, we design two self-supervised tasks for expert and question respectively during the pre-training stage.

Expert Contrastive Task. This task teaches the model to discriminate the representation of a particular expert from other experts. Given j-th expert sequence, we have obtained the expert representation ej (generated from [ER]). The augmented ones (randomly selected from behavior crop or reorder) are considered as positives (with the representation as ¯ej or ˆej), while other in-batch (with B is batch size) experts are considered as negatives. The expert-view self-supervised loss can be formally presented as follows:

where τ is a temperature parameter. Since batches are constructed randomly, the in-batch negative augmented instances ˆej ′ would contain some experts from multiple domains.

Question Contrastive Task. The object is to discriminate the representation of a particular question from other questions, which is similar to the expert contrastive task. Given i-th question, the question-view self-supervised loss can be denoted as:

where ˆqi is i-th question augmented representation and ˆqi′ is the negative augmented instance.

Furthermore, for helping the model capture the CQA language knowledge, inspired by BERT [Devlin et al., 2018], we simply mask 15% of the input tokens randomly (default in the BERT) and then predict those masked tokens. The formal definition is:

where N is the masked token set, V is the vocabulary.

Model Learning. At the pre-training stage, we leverage a multi-task training strategy to jointly pre-train the model:

4.4 Fine-Tuning for Expert Finding

We initialize the expert and question encoders with pretrained parameters. Expert sequences begin with an [ER] token and question sequences with a [QR] token. Their representations are concatenated to predict the matching score sc. Using negative sampling [Huang et al., 2013] with K negative instances, we fine-tune with cross-entropy loss:

where ˆ sc is the ground truth label and sc is the normalized predicted probability.

5.2 Performance Comparison

We compare SSEF with baseline methods on six CQA datasets. The results are reported in Table 1, and we can find:

Some earlier methods (e.g., Doc2Vec, CNTN) obtain poorer results on almost datasets, the reason may be that they usually employ max or mean operation on histories to model expert, which omits different history importance. NeRank utilizes the heterogeneous network embedding to unify raisers, answerers, and questions to learn embedding, which has obtained superior performance. Furthermore, the recent method PMEF has obtained superior performance compared with other baselines. One possible reason is that PMEF models the expert and question under a multi-granularity paradigm, which could be more comprehensive.

Compared with most baselines, SSEF performs mostly better than them on six datasets, which could verify the effective-

5 Experiments

5.1 Experimental Settings

Datasets and Metrics. For pre-training, we construct a corpus from StackExchange 2 comprising 103005 expert and 286835 question sequences, with 670224 interactions across 13262 tags. The pre-training corpus strictly excludes all finetuning validation and test data. For fine-tuning, we evaluate on six domains: Biology, CodeGolf, Gis, Es, Electronics, and CodeReview. Following [Peng et al., 2022], the provider of the "accepted answer" serves as the ground truth expert, while platforms lacking explicit accepted answers are excluded [Li et al., 2019]. As summarized in Table 3, we chronologically split each dataset into 80/10/10% for training, validation, and testing. Performance is evaluated using Mean Reciprocal Rank (MRR), Precision@K (K = 1, 3), and NDCG@20.

Baselines. We compare SSEF with eight competitive baselines: (1) Traditional Methods: NeRank [Li et al., 2019], UserEmb [Ghasemi et al., 2021], ENEF [Liu et al., 2022a] and CGEF [Peng et al., 2024]; (2) PLM-based Methods: TCQR [Zhang et al., 2020], RMRN [Fu et al., 2020], PMEF [Peng et al., 2022], ExpertBert [Liu et al., 2022b], and ExpertPLM [Peng and Liu, 2022]. Furthermore, we test the model transferable performance on PMEF [Peng et al., 2022] and MATER [Zahedi et al., 2024] in Section 5.5.

2 https://archive.org/details/stackexchange

ness of our proposed method. Different from those methods, we carefully construct expert-/question-view input information and corresponding training tasks, which could model expert/question generally in a self-supervised way and alleviate the supervised signal sparsity issue. It is noted that the very recent works ExpertBert and ExpertPLM also achieve superior performance. Our method outperforms them. The reason may be that they only focus on expert modeling and omit the fact that supervised data is sparse in expert finding, while SSEF carefully designs data augmentation strategies and training tasks for self-supervised pre-training, which could learn more precise representations of experts/questions.

5.3 Zero-Shot Performance

Considering our method is a pre-training method, we test the model performance with zero-shot setting. We only fine-tune the pre-trained model on the smallest dataset Biology, and test the performance on the Es and Electronics. The experimental results are shown in Figure 3. It is surprising that the model can still achieve good performance (especially in P@3) by simply transferring from Biology to Es and Electronics without any fine-tuning. This could be attributed to the model has already captured a large amount of expert and question related knowledge during pre-training. And merely fine-tuning on smaller datasets (e.g., biology) is required to stimulate model understanding of the expert finding task, leading to zero-shot performance on other large datasets.

5.4 Ablation Study

Effect of Pre-Training Tasks

To evaluate the effectiveness of our designed inputs and tasks, we compare SSEF against two ablated variants and a standard baseline: (a) Only QR: Employs only question-view information and tasks for pre-training. (b) Only ER: Employs only expert-view information and tasks for pre-training. (c) Original BERT (Baseline): Directly adopts original BERT weights and tasks for further pre-training on the CQA corpus (one question per line), followed by fine-tuning.

As shown in Table 2, we can have the following observations: 1) Original BERT obtains the worst performance. This is equivalent to removing the question-/expert-view tasks from SSEF during pre-training. In this way, the model could lose the ability to model experts/questions. 2) It is surprising that the simple Only QR still outperforms the most of baselines and Original BERT. This is because Only QR could capture CQA language knowledge and benefit question-view representation learning during pre-training, which could help downstream expert finding tasks. 3) Only ER outperforms Only QR. This is because Only ER could pre-train expertview representations that help model experts in downstream tasks. Furthermore, despite the absence of question-view sequences and tasks, Only ER allows the model to simply pre-train question representations due to the question sequences existing in the expert-view sequence. 4) Our complete model SSEF obtains the best results. The reason is that the SSEF could simultaneously pre-train the representations of expert/question with self-supervised tasks, which allows the model to capture unique representations for expert/question before performing the downstream task.

Effect of Different Data Augmentations

We analyze SSEF with different augmentation strategies. ECrop or E-Reorder represents an enhanced view of the expert generated with only crop or reorder operation. EMixture represents randomly selecting one operation (crop or reorder) for generating the enhanced view. The question side setting is similar. As shown in Figure 4, we observe: 1) All strategies outperform the baseline PMEF, indicating they all assist in pre-training. 2) The composition of strategies might not exceed a single strategy, likely because it is hard for the pre-training model to distinguish which augmentation method transformed the input data. 3) E-Reorder usually performs better than E-Crop, whereas the question side is the opposite. This may be because cropping directly destroys the original expert behavior sequence, providing limited knowledge for downstream tasks. Finally, we employ the mixture strategy to construct different augmented views.

Effect of Different Augmentation Ratio

We explore different augmentation ratios (expert parameters αb/βb and question parameters αw/βw varied in {0.1, 0.15, 0.2, 0.25, 0.3}) on CodeReview and Biology datasets. As shown in Figure 5, we observe: (1) The performance increases first and then decreases. This is because increasing the ratio improves robustness, while excessive augmentation seriously damages original information. (2) The turning points of the expert and question curves are different. The reason may be that, compared with the average words per question, each expert has more behaviors and higher tolerance for data augmentation (e.g., 25.7 behaviors vs. 10.6 words in CodeReview). Thus, we set the question parameters (αw, βw) to 0.15 and expert parameters (αb, βb) to 0.2.

5.5 Benefit fromPre-training

In this section, we further conduct detailed experiments to analyze the effect of our self-supervised pre-training model.

Effect of Training Data Ratio in Fine-Tuning

To investigate whether SSEF alleviates data sparsity, we train on varying proportions ([100%, 80%, 60%, 40%, 20%]) of the CodeReview and Biology datasets, keeping validation and testing sets fixed at 10%. As shown in Figure 6, the performance gap between SSEF and PMEF widens as training data decreases. This suggests that SSEF's pre-training learns general representations, effectively reducing dependency on fine-tuning data. Furthermore, this gap is more pronounced on CodeReview than Biology, aligning with CodeReview's inherently higher sparsity. Overall, results confirm SSEF's potential to mitigate sparse supervised signals.

Applying the Pre-Training Strategy on Other Models

Our designed pre-training strategy could learn more general expert and question representations, which have the potential to be applicable for other approaches, e.g., PMEF [Peng et al., 2022], MATER [Zahedi et al., 2024]. We directly add the frozen pre-training expert/question representations to the PMEF and MATER, named PMEF+ and MATER+ respectively for experiments. The results on the public dataset Biol-

X

X

Figure 8: Expert representation 2D visualization on CodeReview dataset: PMEF (left); SSEF (right).

ogy are reported in Figure 7.We can observe that the embeddings derived from our proposed method could consistently improve the performance of PMEF and MATER, which further shows the effectiveness of the proposed method.

Understanding the Pre-Trained Representations.

Recently, uniformity has been recognized as a key property for measuring the quality of representations [Wang and Isola, 2020]. To understand the quality of pre-trained representations, we project the expert representation into 2D space with PCA for visualization. From Figure 8, compared with SSEF, we could observe that most of the expert representations learned by PMEF are mapped in a sharp corner space (the small left part). This is because SSEF pre-trains the expert representation under a self-supervised paradigm. The model could more clearly distinguish experts and prevent them from clustering in the representation space. To some extent, more uniform representations could provide more prior knowledge for the downstream task and promote the performance.

6 Conclusion

In this paper, we propose SSEF, a dual-view self-supervised pre-training language model for expert finding. Different from existing supervised-based expert finding methods, the core of our method is to design a dual-view pre-training model for learning transferable expert and question representations in a self-supervised way. In particular, we propose a model-agnostic framework SSEF to supplement the supervised expert finding task. We devise two types of data augmentation on question-view and expert-view information to construct auxiliary question/expert-view contrastive tasks, which could learn more universal expert/question representations for expert finding. Experiments on six real-world CQA datasets validate the effectiveness and transferability of our pre-training paradigm.