Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialogue

1 Introduction

The design of the dialogue systems aspires to generate consistent, controllable, and diverse responses based on dialogue history [Liu et al., 2022b; Lu et al., 2023]. Initial researches on dialogue systems are often constrained by

∗Corresponding authors.

· · · · · ·

the length of the input window, with a common assumption that responses are primarily influenced by the last one or a few last turns of the dialogue history [Wei et al., 2021; Liu et al., 2022a]. Recently, with the development of large language models (LLMs), researchers propose long-term dialogue tasks and explore the use of rich information in dialogue history for response generation [Xu et al., 2022b].

Despite the impressive achievements of LLMs in opendomain dialogues, capturing causal relationships within extensive dialogue history remains challenging [Feng et al., 2023]. Some researchers identify that LLMs suffer from severe position bias, focusing only on the final context and disregarding historical information [Liu et al., 2023a; Liu et al., 2023b]. This bias leads models to concentrate on spurious position correlations instead of causally relevant utterances, resulting in irrelevant and generic response generation. As shown in Figure 1, Llama2's attention in the last turn is significantly higher than in other turns, neglecting the relevance utterance in the 5-th turn and resulting in the generation of context-irrelevant and generic responses. To address the above problems, some studies aim to compress long-term dialogue history using retrieval-based [Feng et al., 2023] or summary-based [Wang et al., 2023; Lee et al., 2023] methods. Although these works achieve certain results, none substantially improve the ability of LLMs to perceive genuinely causally relevant utterances.

To eliminate position bias and enhance the perception of causal relationships in long-term dialogue, two significant challenges must be addressed: (1) Lack of large-scale dialogue datasets with causally relevant utterance annotations. Perturbation-based causal variable discovery methods assume that models can effectively utilize all input variables. They consider the difference in potential outcomes when binary intervention (presence or exclusion) acts on the variable as the treatment effect. However, the inherent position bias in LLMs hinders their ability to fully leverage dialogue history, making perturbation-based methods unable to be directly used to extract causally relevant utterances. (2) Traditional text generation loss is insufficient in guiding models to eliminate spurious correlations between relevant utterances and position distributions. Models tend to learn imbalances in the distribution of relevant utterance positions in datasets. Existing position debiasing methods primarily disrupt positions, posing challenges in maintaining the consistency of semantic structures when applied to dialogue scenarios.

To address the aforementioned challenges, we propose a model-free Causal Perception long-term Dialogue framework, named CPD. Firstly, we extract relevant utterances for two widely used long-term dialogue datasets using causal perturbation. Analyzing the impact of position bias on the model's causal perception, we introduce a local-position awareness method to mitigate this bias by eliminating intersentence position information. Through sentence-level perturbations on dialogues, we regard the change in the perplexity [Horgan, 1995] before and after the perturbation as the treatment effect of the perturbed utterance. Validation of sentence-level conditional independence in sets of causally relevant utterances, followed by the application of clustering algorithms to categorize dialogue history into causally relevant or irrelevant parts. Secondly, inspired by invariance learning [Chang et al., 2020], the core of preventing models from fitting position bias is to direct models to concentrate on invariant causal variables. We argue that responses and their corresponding causally relevant utterances should exhibit consistency under perturbations. Specifically, we conduct sentence-level perturbations separately on causally relevant and non-causally relevant parts, encouraging the model's generation to be either consistent or inconsistent with gold responses based on whether the causal variables are perturbed or not. A positional difference sampling strategy, based on the probability of relevant utterance positions, is employed to address the imbalance of relevant utterance positions without compromising the integrity of the dialogue structure.

The contributions of this paper are listed as follows:

• We propose a relevant utterance extraction method based on causal perturbation, which can avoid the interference caused by the position bias of LLMs through localposition awareness.
• We propose a causal perception fine-tuning strategy that
• effectively alleviates the model's position bias and enhances its ability to perceive correlations in dialogues.
• Experimental results on two benchmark datasets demonstrate that our proposed method can consistently outperform the state-of-the-art baselines in terms of objective and subjective evaluation metrics.

2 RelatedWork

2.1 Position Bias and Long-TermDialogue

Position bias in language models has undergone extensive examination. Ko et al.[2020] observed that language models introduced position bias due to the imbalance position distribution of relevant sentences in datasets, relying on fixedposition utterances instead of real causal correlation during inference. Wang et al.[2021] and Liu et al.[2023a] respectively confirmed the existence of position bias in a variety of pre-trained models utilizing different position embeddings and a variety of LLMs of different sizes.

Existing position debiasing methods primarily involved disrupting the position information in datasets and constraining the consistency of the original model during the finetuning process. Ko et al.[2020] and RPP [Amor et al., 2023] applied random position perturbations at the document and word level separately. Some other works divide context into short segments [Ivgi et al., 2023; Li et al., 2023], or reorder input context based on relevance [Peysakhovich and Lerer, 2023; He et al., 2023]. However, dialogues exhibit temporal structure, and position perturbation can destroy dialogue structures, leading to a substantial decrease in comprehension. ZOE [Liu et al., 2024] required the model to fit both task labels and original model output, alleviating the impact of position imbalance by enforcing consistency between the fine-tuned model and the original model. Unfortunately, in dialogue scenarios, the inherent position bias of the model aligns with the position imbalance in the data, causing the method to fail to a certain extent.

To explore the model's ability to comprehend rich semantics in long-term dialogue history, Xu et al. [2022a] constructed a multi-turn long-term dialogue dataset. Existing works primarily focused on enabling LLMs to acquire longterm dialogue awareness by compressing dialogue history to alleviate position bias. RSM [Wang et al., 2023] and Lee et al.[2023] employed LLMs to summarize dialogue history as an external memory pool, enhancing LLMs with long-term memory capabilities. CONSTRAIN [Feng et al., 2023] assumed that only two sentences in the dialogue history were relevant to the response, utilizing a trained language model to retrieve relevant utterances. However, these methods suffer from dual challenges of information loss during compression and a lack of substantial improvement in the model's inherent ability to comprehend extended contexts.

2.2 Causal Inference in NLP

Causal inference is a method used to determine whether correlations in data truly reflect cause-and-effect relationships [Pearl et al., 2016; Alaa and Van Der Schaar, 2019]. Works on causal inference in NLP mainly focused on word granular detoxification in pre-trained language models. For instance, Abraham et al.[2022] and Madhavan et al.[2023] both engaged in word-level language model detoxification, employing average treatment effect and counterfactual enhancement. Wang et al.[2023] is similar to our work. They annotate a small test set of causally relevant utterances, named CGDIALOG, and discovered that the causal perception ability of language models is deficient, meaning that language models exhibit similar outcomes to perturbations of causally relevant or non-causally relevant utterances. They simply attributed this problem to overfitting. In our investigation, we delved further and identified position bias as the root cause of this phenomenon.

3 Methodology

3.1 Causal View for Response Generation

Structure Causal Model (SCM) [Shanmugam, 2001] is a method of describing causal correlations among multiple variables. In this paper, we describe the causal structure in dialogue as a causal graph composed of four variables: input dialogue D, response R, causal part C, and non-causal part S. Figure 2 illustrates the SCM of dialogue.

Dialogue D consists of a causal part C and a non-causal part S of the response (C → D ← S), where the causal part C is the minimum set of corresponding endogenous cause utterances leading to the response R (C → R). An outstanding language model should exhibit sensitivity to causal part C while maintaining robustness to non-causal part S. Unfortunately, due to the probabilistic correlation between the causal part C and the non-causal part S (C ⇠----⇢ S), including the imbalanced distribution of causal part utterances across positions, often leads models to overfit position information, fixate on utterances at nearby positions, and lose the ability to perceive causality (S --⇢ R).

Treatment Effect (TE) [Austin, 2011] quantifies the impact of a variable on the target outcome. In practice, TE is often assessed through the conditional independence assumption [Dawid, 1980] and counterfactual reasoning [Rafetseder et al., 2013]. For treatment variable ui, a binary treatment is employed to evaluate the treatment effect of the outcome. The TE of utterance ui in dialogue D is defined as:

where D\ui represents the dialogue when utterance ui is disturbed, f(·) is the perplexity [Horgan, 1995] of the language model to generate correct responses. To eliminate the potential impact of changes in other co-variables after the utterance absence, we use meaningless utterances of similar lengths, such as 'hello,' 'thank you,' etc., to replace ui to construct the counterfactual condition of the absence treatment.

3.2 Causal Identification and Position Bias

Outstanding language models should adeptly identify causal correlations in dialogues. Specifically, the TE of causally relevant utterances should be significantly higher than the TE of non-causal utterances. Unfortunately, Abraham et al.[2022] observe that language models lack this identification ability. We further observe that the model's causal identification ability is strongly tied to the position of causally relevant utterances in the dialogue. The normalized treatment effect TEreg(ui) is employed to measure the causal correlation between utterances and responses.

As shown in Figure 3 (a) and (b), we find that: (1) The LLMs can only identify the causal correlation in the last 1 and 2 turns of dialogue. (2) Irrespective of whether the utterances are relevant or not, LLMs consistently exhibit greater sensitivity to perturbations in the last few turns of dialogues. (3) Despite fine-tuning on domain data enhancing the model's sensitivity to causally relevant utterances, the accurate distinction remains challenging in longer dialogue histories.

This indicates that LLMs have the potential to distinguish between relevant and irrelevant utterances. One way to eliminate the influence of position information is to remove all position embedding, as shown in Figure 3 (c). After removing position embeddings, the model no longer focuses on position-specific utterances. Unfortunately, it also results in the damage of semantic information, rendering the LLMs incapable of identifying relevant utterances.

3.3 Causally Relevant Utterance Extraction

To achieve a balance between semantic information and position bias, we propose a sentence-level local-position awareness method for each layer of LLMs. Specifically, we restrict position information within sentences, and inter-sentence attention only uses semantic correlations. To ensure the method's applicability to models employing different positional embeddings, we directly modify the attention matrix. The model utilizes attention with position embeddings Ape t,s when the input words are in the same sentence. In contrast, when the input words are not in the same sentence, the attention is without position embeddings ˜Ape t,s :

where xt and xs denote the t-th and s-th input word, mpe A and ˜mpe A are the average values of attention matrices Ape and ˜Ape respectively, used to balance the difference between two attention matrices.

After fine-tuning, the model's performance is shown in Figure 3 (d). It can be observed that our method effectively mitigates position bias, leading to improved identification of causally relevant utterances. To extract the minimal causally relevant utterance set in dialogues, we individually measure the TE of each utterance in the dialogue, denoted as [TE(u1), TE(u2), ..., TE(u|D|)], where |D| represents the number of dialogue turns. Due to the difference in TE between causally relevant and non-causally relevant utterances, we employ a simple clustering algorithm, K-means [Krishna and Murty, 1999], to obtain the causally relevant C and the non-causally relevant utterance set S. The initial clustering centers of the two sets are initialized as the minimum and median of the input data, respectively.

After verifying the effectiveness of our method with the 88.6% precision on the CGDIALOG test set, we extract relevant utterances from two long-term dialogue datasets, ESConv [Liu et al., 2021] and MSC [Xu et al., 2022a]. We calculate the position distribution Q of causally relevant utterances, where qi ∈ Q represents the frequency of causally relevant utterances with the i-th turn distance to the response. A severe imbalance is observed in the position distribution of causally relevant utterances, which might be the cause of model position bias.

3.4 Causal Perception Finetuning

The fine-tuning of LLMs follows the paradigm of instruction fine-tuning, where instructions and dialogue D are concatenated and fed into the model for generating responses R.

During the fine-tuning process, our objective is for the model to acquire domain knowledge from the data while being sensitive to causal correlations. The loss during the finetuning process is divided into two parts: prediction loss and causal perception loss. The prediction loss ensures that the model accurately generates the gold responses and captures domain knowledge in the training data. The causal perception loss is employed to enhance the model's sensitivity to causal correlations.

where α and β are used to scale the three losses to similar orders of magnitude. The prediction loss is to maximize the cross-entropy between model predictions and gold responses.

For the sake of brevity, the instruction in the formula is omitted following. The causal perception loss is designed to require the model to focus on the causally relevant utterances while being sensitive to perturbations in environmental variables, such as non-relevant utterances. The causal perception loss consists of two parts: invariant risk minimization LIRM and maximizing treatment effect LMTE.

Invariant risk minimization (IRM) [Arjovsky et al., 2019] drives the model to grasp causal invariance across environments, that is, the model's outcomes should remain consistent across various environments, which are constructed by perturbing non-causally relevant utterances. For dialogue D = {u1, u2, . . . , u|D|} consisting of multiple utterances, we construct counterfactual dialogue D \ ui in which noncausally relevant utterances ui ∈ S are replaced. To ensure that the replacements do not significantly alter the semantic structure of dialogues, we randomly select utterances for substitution from non-causally relevant utterances in other dialogues. Invariant risk minimization loss minimizes the KL divergence of responses generated by counterfactual and original dialogues.

where sg means stopping gradient during backpropagation.

Maximizing treatment effect (MTE) [Marshall et al., 2005] aims to help the model learn the consistency relationship between responses and corresponding causal variables. When causally relevant utterances are replaced with those from other dialogues, the loss expects models to generate utterances that exhibit maximum dissimilarity compared to the original dialogue model output.

Sampling strategy. To overcome the imbalance position distribution of relevant utterances, we employ a sampling strategy that involves enforcing more perturbations at positions with the low likelihood of being a relevant utterance in the dataset. At the dialogue level, multiple perturbations are performed for each dialogue when calculating causal perception loss. The number of times n for both auxiliary tasks IRM and MTE is determined through the following formula:

where ⌊·⌋ denotes rounding down, | · | represents the number of elements in the set, and q|D|-i represents the frequency of the i-th utterance in the dataset being a causally relevant utterance.

At the utterance level, each time the auxiliary task is calculated, the probability S(ui) of perturbing each sentence ui is different. For IRM and MTE, the probability that utterance ui is perturbed is,

4 Experiments

4.1 Datasets

To evaluate the effectiveness of our proposed method, following previous works [Wang et al., 2023; Feng et al., 2023], we conduct experiments on two widely used benchmark datasets, ESConv [Liu et al., 2021] and MSC [Xu et al., 2022a], for long-term dialogue. We use the same data preprocessing and train/valid/test splitting strategy as in [Feng et al., 2023].

4.2 Baselines

To demonstrate the effectiveness of our proposed method, we compare it with three kinds of baselines: (1) Raw and finetuned LLMs have outstanding performance in open-domain dialogue tasks. The model can improve its performance in the tasks during fine-tuning in the corresponding task domain. (2) Long-term dialogue methods. RSM [Wang et al., 2023] continuously summarizes long-term dialogues and uses the summary as external memory to alleviate the memory forgetting of long-term dialogues in LLMs. CONSTRAIN [Feng et al., 2023] assumes that except for the last sentence in the dialogues, there is only one sentence relevant to the response. Relevant utterances in dialogue history are retrieved through a trained language model and concatenated with the last sentence as input for response generation. (3) Position debiasing methods. RPP [Amor et al., 2023] is extended to sentence-level position random perturbation to ensure that the training data no longer has imbalances in position distribution. ZOE [Liu et al., 2024] fits both gold response and the suboptimal response generated by the original model to enforce consistency between the fine-tuned and original model. For the fairness of the experiments, the backbones in all baselines are replaced by two widely used LLMs, Llama2-7B-chat and Qwen-14B-chat.

4.3 EvaluationMetrics

Automatic Evaluation. (1) Word Overlap. We report BLEU-n (n=1, 2) [Papineni et al., 2002] and ROUGE-L [Lin and Och, 2004] to evaluate the coherence and word overlap of generated utterances. (2) Diversity. We employ Distinctn (n=1, 2) [Li et al., 2016] to evaluate the diversity of the generated response.

Human Evaluation. We adopt Relevance, Fluency, and Informativeness of the generated utterances with the rating range of [0, 2]. We recruit three experienced annotators to evaluate 100 randomly selected dialogues with a length of more than 20 turns. The Fleiss Kappa is 0.72, indicating consistency in the estimates of annotators.

4.4 Implementation Details

Throughout the experiments, we use Adam optimizer [Kingma and Ba, 2015] with 3e-4 initial learning rate and the 128 batch size. All methods are trained for up to 12 epochs. To improve experimental efficiency, we use lora [Hu et al., 2021] with rank 32 to fine-tune large language models. Both training and inference use 4-bit weight quantization by bitsandbytes [Dettmers et al., 2022].

4.5 Main Result

The main evaluation results are shown in Tabel 1.

Automatic Evaluation. CPD attains optimal performance by effectively mitigating the model's position bias and enhancing its causal perception ability. Long-term dialogue methods alleviate the poison of position bias by compressing long-term dialogue history through summarization and retrieval, respectively. Notably, in the ESConv dataset with shorter dialogue turns, summary-based RSM outperforms CONSTRAIN, while retrieval-based CONSTRAIN excels in the longer MSC dataset. These phenomenons underscore that while iterative summarization may retain more information in the short term, it leads to inevitable information loss with multiple iterations. CONSTRAIN neglects the potential significance of multiple causally relevant historical utterances, exhibiting suboptimal performance for shorter dialogues. Additionally, CONSTRAIN performs poorly in diversity indicators, as it overlooks a significant portion of the dialogues' information. Unfortunately, the aforementioned methods, while effectively enhancing LLMs by reducing dialogue history length, fall short of fundamentally improving the models' ability to comprehend extended contexts. Position debiasing methods primarily focus on eliminating position bias but do not address the underlying issue of causal perception in LLMs, resulting in suboptimal performance. RPP, destructing the dialogue structure during fine-tuning, seriously impairs the model's perceptual ability, leading to the poorest performance in word overlap metrics.

Human Evaluation. Overall, the results of manual and automated assessments align consistently. All methods achieved similar performance in fluency metrics, showcasing the remarkable capabilities of LLMs. The raw and finetuned models underperform in relevance and informativeness metrics due to position bias. CPD enhances both relevance and information content indicators, achieving an average improvement of 5% and 7%, respectively, compared to the state-of-the-art baselines. The significant improvement underscores the efficacy of our method in effectively mitigating position bias in LLMs for specific tasks through causal perception finetuning.

4.6 Ablation Study

To demonstrate the effectiveness of our proposed method, we establish three variants: (1) w/o IRM: remove the invariant risk minimization loss (Equation 7); (2) w/o MTE: remove the maximizing treatment effects loss (Equation 8); (3) w/o Dialogue

. . .

Assistant : Hi, do have any issues that you would like to share with me today?

User : I am in disputed mod with my friends

Assistant : I am sorry to hear that. So, you and your friends are not seeing eye to eye? Do you mind telling me what the issue is that you are not agreeing on?

. . .

User : They need more money from me, but i couldn't give that much amount

. . .

User : Sure, it is my new way of relationship to me with others

Assistant : Yes, you're growing and trying to do your best and wanting to expand your boundaries and make them clear. You have taken the right approach and are on the best track to having a successful relationship in the future. Great Job! Thank you for sharing your story with me today.

User : What kind of people do I dispute with?

Assistant :

sampling: remove the positional difference sampling strategy. The experimental results are shown in Table 2.

Experimental results indicate that all ablation variants experienced varying degrees of performance degradation, demonstrating the effectiveness of the components in our method. Furthermore, invariant risk minimization loss can effectively make the model's generation towards gold responses, while maximizing treatment effects loss enhances the diversity of model generation responses. The performance of removing sampling strategy is extremely degraded, proving that it effectively alleviates position imbalance.

4.7 Case Study

Table 3 illustrates an example. The existence of position bias in both the raw and finetuned LLMs leads to generating safe and general but context-irrelevant responses. RSM maintains part of memory capabilities through iterative summarization. CONSTRAIN, despite finding a partially correct answer through retrieval, suffers from substantial information loss in the dialogue, leading to a notable decrease in response diversity. RPP disrupts the dialogue structure by altering sentence-level positions during fine-tuning, impairs the model's ability to identify causally relevant utterances, and results in generating context-irrelevant responses. ZOE eliminates part of position bias of LLMs, but the model's causal perception ability is still weak and unable to identify all relevant utterances for response generation. In contrast, CPD benefits from targeted causal perception fine-tuning, effectively perceiving the correct answer and providing informative answers while considering the entire dialogue history.

5 Conclusion

In this paper, we analyze the deleterious impact effects of position bias in LLMs on long-term dialogue tasks from a causal perspective. To solve the problem, we propose a model-free Causal Perception long-term Dialogue framework (CPD). We extract causally relevant utterances and mitigate position bias through causally perturbed fine-tuning. Specifically, we propose local-position awareness by localizing position information within utterances and further combining it with a perturbation-based method to extract causally relevant utterances. We also propose a causal perception fine-tuning strategy that guides models to focus on causal invariant variables by differently perturbing causally relevant and non-causally relevant utterances in dialogues. A positional difference sampling strategy is employed to address positional imbalances in datasets while maintaining the temporal structure of dialogues. Experiment results demonstrate the effectiveness of our method in alleviating position bias, resulting in informative and human-like response generation.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110, No. 62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.