Beyond What If: Advancing Counterfactual Text Generation with Structural Causal Modeling

1 Introduction

Causal inference has always been a hot research topic in the field of Natural Language Processing (NLP), achieving significant results [Luo et al., 2016; Feder et al., 2022; Hu and Li, 2021]. In recent years, the study of text generation using counterfactuals has gradually gained traction [Qin et al., 2019; Hao et al., 2021]. This research avenue, anchored in conditional text generation, delves into the exploration of generated text in counterfactual worlds. Here, the text outcomes are examined in light of altered conditions, contrasting them with those under the original, actual-world conditions. One illustrating example of counterfactual story generation is given in Figure 1. Each text segment comprises three parts: background (marked as B), condition (marked as C), and outcome (marked as E). From the original text on the left, we learn that the Vice Premier is visiting the United States (B), with both parties seeking to prevent economic decoupling (C), leading to a meeting outcome where both sides welcome the establishment of a China-US economic working group, indicating a positive result (E). In this context, the counterfactual condition or background is the opposite of the original scenario. For example, the counterfactual condition could change from "reached an agreement" to "fail to reach an agreement", leading to "hindering bilateral and global economic relations", an outcome that completely contradicts the original.

∗Corresponding author. Email: zhangxiaofeng@hit.edu.cn

Recent advancements in counterfactual text generation, moving beyond mere data augmentation, have led to innovative developments. For example, Qin et al. [Qin et al., 2019] introduced an open-source dataset that has become essential for evaluating model performance in this field. Their approach reframed the issue as a sequence-to-sequence generation problem, where background (B), altered condition (counterfactual C), and outcome (E) are concatenated as inputs for the GPT-2 model, generating counterfactual endings. Further, [Qin et al., 2020] explored unsupervised generation of story endings, incorporating both past and future contexts for controlled generation. However, these methods primarily focus on generating endings under given counterfactual conditions, often overlooking the causal impact of the background on the conditions and consequently, the ending.

Addressing this gap, our work aims to master this intricate task by allowing modifications in both background and conditions for hierarchical dependent causal text generation. For instance, altering the original background to "due to scheduling conflicts, the meeting is cancelled", as depicted in the right side of Figure 1, necessitates a change in the counterfactual condition to "missed the opportunity to reach an agreement", leading to a minimally edited ending like "increasing market uncertainty". To tackle this challenge, we propose a novel approach based on structural causal models (SCM) [Pearl et al., 2016]. This involves adapting SCMs to our context and integrating them with a pre-trained model to generate both counterfactual C and E. A unique disentanglement component is designed to unravel the hierarchical causal relationships among B, C, and E, ensuring the separate influence of each variable in the underlying causal model. The disentangled variables in the latent space are then strategically combined to form the decoder's input, producing the counterfactual endings.

The major contributions of this paper are summarized below:

• We evolve the concept of counterfactual text generation into a hierarchical dependent causal text generation problem. This is achieved by enabling counterfactual modifications in both background and condition elements, and by developing an end-to-end approach based on structural causal models (SCM).
• A novel disentanglement component is introduced, which effectively maps multiple hierarchical dependent variables into a latent space. We also adapt and retrain a BART model, integrating it with our SCM component, specifically tailored for our task.
• We conduct comprehensive experiments not only on a public counterfactual story generation dataset but also on a custom-constructed dataset in the financial domain. Our experimental results affirm the superiority of our approach over existing methods, as evidenced by various automatic and human evaluation criteria.

2 RelatedWork

The task of counterfactual text generation, a subset of conditional text generation, has been explored in various contexts including context-based [Voita et al., 2018], personalized [Lu et al., 2017], and topic-based text generation [Yang et al., 2021]. Despite the advancements, most approaches overlook causal reasoning, crucial for varying conditions [Roese, 1997;

Sch¨olkopf et al., 2021; Fern and Pope, 2021; Wang et al., 2019; Qin et al., 2019; Hao et al., 2021; Chen et al., 2022; Zellers et al., 2019]. Incorporating Structural Causal Models (SCM) [Pearl et al., 2016], researchers have enhanced text generation with causal reasoning abilities. Examples include augmenting data for neural machine translation [Liu et al., 2021], controllable text generation [Hu and Li, 2021], and counterfactual text optimization [Fern and Pope, 2021]. Various counterfactual generation methods have been proposed, employing standard sequence-to-sequence models and GPT2 [Qin et al., 2019; Hao et al., 2021; Chen et al., 2022], but they heavily rely on the generation capabilities of pretrained models, which can misidentify counterfactual content [Zellers et al., 2019]. Disentangled representation learning, crucial for interpretable and independent data variation, has been applied across sequential data analysis [Yamada et al., 2019], weakly-supervised learning [Zhu et al., 2023], high-fidelity synthesis [Lee et al., 2020], and various NLP tasks [Vasilakes et al., 2022; Dang-Nhu, 2021; Wang et al., 2019]. Our work aligns with studies like [Ren et al., 2022; Khrulkov et al., 2021] that focus on modeling variations and extracting disentangled representations, respectively. Our work, therefore, lies at the convergence of conditional text generation, disentangled representation learning, and structural causal models. It pushes the boundaries of these fields to address the complex challenge of hierarchical dependent causal text generation.

3 Preliminaries and Task Setup

We consider a textual segment T = {B,C,E}, where B is the background context, C is a specific condition, and E is the consequent outcome. This structure, suitable for a wide range of domains, allows for the exploration of complex, multilayered causal relationships. In counterfactual text generation, we aim to modify B or C with counterfactual content, leading to a new outcome E′ that should be contextually coherent yet minimally edited. This process unveils the intricate causal layers within the text. The task is bifurcated into two primary sub-tasks:

Task 1. Intervening Background. Given T and a counterfactual background B′ , the task involves generating a condition C ′ and an outcome E′ that logically follow the new background, creating T ′ = {B′ ,C ′ ,E′ }. This task emphasizes the exploration of causal layers from background to condition and outcome. The generation of E′ encompasses multilayered causal reasoning:

Task 2. Intervening Condition. Given T and a counterfactual condition C ′ , this task focuses on generating a new outcome E′ consistent with C ′ . The updated segment is T ′ = {B,C ′ ,E′ }, reflecting a different causal layer intervention. The process is formulated to capture the cascading effects of changing conditions on the outcome:

These tasks underscore our method's capacity to navigate and manipulate the multi-layered causal relationships inherent in text, showcasing its applicability in generating contextually rich and causally coherent counterfactual scenarios across diverse textual domains.

By minimizing Eq. 1, most of existing content of E could be preserved. If further minimize Eq. 2, U will be minimized as V is simultaneously trained to maximally generate X. Thus, the variance of U is under a well control. To guarantee the independence between U and V , we adopt mutual information to measure the statistical dependence between two variables, its general form could be written as

where H is the Shannon entropy. However, this equation is intractable and thus a lower bound on I(U, V ) is desired to estimate. By considering the KL divergence as its distance, we can derive its supremum according to [Belghazi et al., 2018], given as

where Tθ : U × V → R is a transformation function parameterized by a neural network. The mutual information loss is then estimated as

Therefore, the overall loss of this disentanglement component could be written as

where α is the coefficient. Note that this component will be executed three times respectively for B, C and E as the input.

4.2 Counterfactual Text Generation Component

To generate the counterfactual text, a structural causal model (SCM) is adapted in this section. First, we briefly review the conventional SCM as follows. It consists of two sets of variables: Endogenous variable En = {En1, En2, ..., Eni} and Exogenous variable Ex = {Ex1,Ex2, ...,Exi}, and a

4 The Proposed Approach

Our proposed approach is designed to generate counterfactual text by intervening at different layers of the background or condition. The approach comprises two primary components: (1) Disentanglement Component: This component disentangles the background, condition, and outcome variables into two types of variables in the latent space: U and V. Here, U symbolizes the unchanged content, aligning with the concept of minimal editing, while V represents the variable content, corresponding to the causal variables. This disentanglement facilitates the exploration and manipulation of multi-layered causal relationships within the text. (2) Counterfactual Text Generation Component: This component employs the Structural Causal Model (SCM) to enhance causal reasoning capabilities. It is seamlessly integrated with a pre-trained model to generate counterfactual text that adheres to the altered background or condition. This integration enables the generation of text that is not only contextually coherent but also reflects the intricate causal dynamics within the text.

The architecture of our approach, which caters to the generation of contextually rich and causally coherent counterfactual scenarios across various textual domains, is depicted in Figure 2.

4.1 Disentanglement Component

We design this component which respectively disentangles the input variables, i.e., B, C, E into each pair of varying variables and unchanged variables. Then the generation is conducted in the latent space with the combination of these latent variables (details will be given in the next subsection). The disentanglement processes are illustrated as follows.

The disentanglement component is built based on the encoder-decoder architecture, as shown in Figure 2. Let's take B as the input X, and the component is to disentangle X into variable U and V residing in the latent space. Let qϕ(U, V |X) and Pω(X|V ) respectively denote the encoder and decoder, variable V is then fed into a decoder DV to reconstruct X,

And another decoder DUV , denoted as Pϵ(X|U, V ), takes variable U and V as the input to reconstruct X, the reconstruction loss of DUV is written as, set of functions f = {fX : WX → X|X ∈ En},WX ⊆ (En ∪ Ex) - {X} that assign values to endogenous variable in En, such that

The exogenous variables are the direct cause of the endogenous variables [Pearl et al., 2016]. To perform interventions on the endogenous variables En, the corresponding counterfactual changes could be described as,

where do-operator assigns an intervention value to the variable. As fEn′i is not an observed value, according to [Pearl et al., 2016], it is assumed that Based on the fundamental law of counterfactuals [Pearl et al., 2016] that future eventualities does not alter the past, we have,

This formulation facilitates our counterfactual reasoning process as we do not need to learn new counterfactual functions for every interventions.

Our proposed causal graph is depicted in Figure 3(a), the corresponding causal relations among these variables are formulated as

The causal inference process of our proposed two sub tasks are illustrated as follows.

Causal Inference by Intervening Background. The causal inference process is plotted in Figure 3(b). As shown in the red arrow, after intervening B to B′ by performing dooperator as do(B = B′ ), the condition C is counterfactually changed to C ′ via a causal function fc, and thus the story endings E′ is causally dependent on B′ and C ′ . The corresponding causal functions are directly given as

Causal Inference by Intervening Condition. The causal inference process by intervening condition C is plotted using red arrow in Figure 3(c). It is obvious that the intervened C ′ is independent of B, and thus E′ relies on C ′ and B, estimated as

Learning Causal Functions. We approximate the causal functions fc and fe using two DNNs, two decoders are employed, respectively denoted as DC and DE.

Our counterfactual text generation task is essentially to answer the question like "Given the whole story, what is value of E if B had been a counterfactual B′ or C had been a counterfactual C ′ ?". The steps to answer the question is given as follows.

• Abduction: Training the network depicted in Figure 2 to acquire the latent representations of the variables as well as the causal functions f approximated by the decoder.
• Action: Intervening B or C to generate the counterfactual embedding VB′ or VC′ via the disentanglement component.
• Prediction: Using the learnt causal function f and exogenous variables shown in Eq. 10 and 11 to infer the counterfactual content.

Model Training. During the model training phase, counterfactual content B′ and C ′ are treated as known variables since the dataset has provided C ′ and B′ is acquired through applying method from [Wu et al., 2021] where they designed certain control codes like "negation" or "delete" for the counterfactual generation and used a fill-in-the-blank structure to specify the position where the perturbation occurs. And thus the VB′ is obtained by inputting B′ to the disentanglement module.

Decoder DC is trained to approximate the causal function fc defined in Eq. 7, calculated as

Decoder DE is trained to approximate the causal function fe defined in Eq. 8, calculated as

Without loss of generality, the cross-entropy loss is adopted as the model loss, formulated as

5 Experiments

In this section, we present a comprehensive overview of our experimental approach. We begin by introducing the datasets used in our experiments, including the newly constructed counterfactual financial text generation dataset. Following this, we outline the evaluation criteria and the models used for comparison purposes. Extensive experiments are performed to answer following research questions:

• RQ 1: Whether the proposed approach is superior to the state-of-the-art counterfactual story generation approaches or not?
• RQ 2: Whether the proposed component, e.g., disentanglement component, works or not? (ablation study).
• RQ 3: What is the quality of the generated counterfactual story (case study)?

5.1 Dataset Construction

The dataset was constructed to validate our model's versatility across various domains; detailed construction details are provided in the appendix A due to page limit.

5.2 Experimental Datasets and Parameter Settings

We use publicly available counterfactual story generation dataset [Qin et al., 2019] and our constructed counterfactual financial text generation dataset to valid our proposed method. The detail of the story dataset and the parameter settings are provided in the appendix B due to page limit.

5.3 Evaluation Criteria

We adopt the automatic and human evaluation criteria to evaluate the model performance of the proposed approach as well as the compared models described as follows.

Automatic Evaluation. A good number of automatic evaluation criteria have been adopted in the experiments. The ROUGE [Lin, 2004] is adopted to evaluate the hit rate, the Word Mover's Similarity (WMS) [Kusner et al., 2015] and the BERTScore [Zhang* et al., 2020] (BERTS) are adopted to evaluate semantic similarities, and we also fine-tune a Bert model on the story dataset which is denoted as 'BERT-FT'. To evaluate the consistency of the generated ending, we adopt a factual consistency metric [Huang et al., 2021] (FactScore) as well as a model-based semantic consistency metric, denoted as 'NSPScore'. To calculate the 'NSPScore' score, we also fine-tune a Bert model for the next sentence prediction (NSP) task using this story dataset.

Human Evaluation. For human evaluation criteria, we randomly select 100 generated samples from testing set and seek 2 groups of annotators where each group contains 3 independent annotators to evaluate the quality of the generated stories. Annotators of group A and group B have similar backgrounds to alleviate human bias. The selected text is anonymized and sent to each group for human evaluation. The average results of these two groups are reported as our final results. The human evaluation criteria is to measure the consistency and similarity of the generated content to the background, condition and original ending. To this end, the human annotators are required to answer following questions by giving scores from 1 to 3, and the higher the score the better the results. These questions are listed as follows.

• BACKGROUND (BG): To what extent is the generated ending consistent with the background? Whether the generated ending conflicts with the background or not.
• CONDITION (CF): Does the generated ending well reflect the counterfactual condition?
• PLOT: Is the generated ending similar to the plot of the original ending?

5.4 Baseline Models

To evaluate the effectiveness of the proposed approach, a number of baseline models [Radford et al., 2019; Lewis et al., 2020; Raffel et al., 2020] and state-of-the-art models [Qin et al., 2020; Chen et al., 2022] are compared. The baseline models are pre-trained models under Zero-shot, Unsupervised and Supervised settings. The details of the comparison models are provided in the appendix C due to page limit.

5.5 Experimental Results

RQ 1: Performance Comparison

Automatic Evaluation Results. The corresponding automatic evaluation results of all approaches are reported in Table 1 and Table 2. From these tables, we have the following observations. First, the proposed model demonstrated outstanding performance, particularly in the aspect of factual accuracy as measured by the FactScore metric. This exceptional result on the FactScore is a testament to the model's proficiency in generating text that is not only coherent and contextually relevant but also factually consistent, a crucial aspect in counterfactual text generation.

Second, the significance of the reconstruction mode (RC) in the model's architecture was profoundly evident in the experimental outcomes. The incorporation of RC led to notable improvements across various models and settings, particularly enhancing the FactScore. This underscores the RC's pivotal role in refining the model's capability to adhere to the principles of minimal editing while adjusting to new counterfactual conditions.

Human Evaluation Results. The human evaluation results for the counterfactual financial and story datasets, presented in Tables 3 and 4, show that the proposed models, particularly Our-large and Our-large+bcf, consistently outperformed baseline models in key aspects. For the financial dataset, Ourlarge+bcf excelled in generating text that aligns well with altered scenarios, especially in Background (BG) and Counterfactual (CF) components. This performance highlights the model's capability in maintaining coherence and factual consistency, aligning with the aim to explore causal reasoning among different text components. The Our-large model also showed strong performance, further emphasizing the effectiveness of the approach in creating contextually relevant counterfactual narratives.

For the story dataset, Our-large+bcf achieved the highest scores, particularly in BG and PLOT dimensions, demonstrating its proficiency in maintaining narrative coherence amidst counterfactual changes. Similarly, Our-large notably excelled in the CF aspect, suggesting its strength in generating semantically consistent counterfactual endings. These results across both datasets underscore the robustness of the proposed approach in counterfactual text generation, affirming its utility in producing complex, coherent, and contextually appropriate narratives in varied domains.

RQ 2: Effectiveness of Different Component

Disentanglement Component. To evaluate the effect of disentanglement component, we visualize the embedding results of UB,UC,UE and VB, VC, VE (before and after training) in Figure 4 for subjective assessment. It can be seen from Figure 4 (a), (c) and (e) that before training, the embeddings of two latent variables interweave with each other. However, after training with the disentanglement component, as plotted in Figure 4 (b), (d) and (f), the embeddings of these variables could be well separated. This verifies the effectiveness of the proposed disentanglement component.

Ablation Study. In the ablation study conducted on both financial and story datasets, the model Our-large and its variant Our-large+BCF demonstrated outstanding performance, particularly in enhancing factual accuracy as evidenced by the FactScore metric. The results, outlined in Table 5 and Table 6, revealed that the removal of key components like the disentanglement component (Ldis) and mutual information loss (LU↔V ) led to significant performance drops, especially in maintaining factual consistency. This underscores the critical role of these components in the model's ability to generate coherent and contextually relevant counterfactual texts. The variant Our-large+BCF showed superior results across various metrics, confirming that interventions in the counterfactual background markedly improve the model's effectiveness in both financial and narrative text generation domains.

RQ 3: A Case Study

In the case study for the counterfactual story generation dataset, we chose to focus on narrative examples rather than financial ones due to their more intuitive nature for illustrating the model's capabilities. The results are reported in Table 2 in the appendix due to page limit. From the table, it is obvious that the zero-shot model generates a quite different story compared with the original story. For the 'BART-base+FT+RC' model, it tends to copy the original ending. However, our model could generate reasonable endings that are dependent on the varying conditions or backgrounds. For the case "intervening the condition", the condition is changed from "He decided to rob a bank" to "He decided to ask the bank teller for some money". The corresponding generated ending is like "He was rejected and held up the bank teller". This ending is causal reasonable as the "held up the bank teller" happened after "he was rejected", and the overall ending satisfies the minimal edit requirement. For the case "intervening the background", the generated text contains "His fellows" corresponding to the background "his work office" and the rest content is minimal edited. This observation verifies the causal reasoning ability of the proposed approach.

6 Conclusion

In this paper, we introduced an approach to counterfactual text generation, utilizing structural causal models (SCM) to navigate complex, multi-layered causal relationships across diverse domains such as narrative stories and financial reports. Our method, which stands out from traditional single causal analysis, focuses on disentangling text components into latent variables for generating causally influenced outcomes. This approach is validated through extensive experiments on both a public story generation dataset and a specially constructed financial dataset, demonstrating superior performance over existing methods.