Disentangling Societal Inequality from Model Biases: Gender Inequality in Divorce Court Proceedings

1 Introduction

The 2011 decennial census in India gave its citizens the following choices to select their marital status - never married, separated, divorced, widowed, married. Based on the census data, a study reported some startling facts [Jacob and Chattopadhyay, 2016]: 1.36 million of the Indian population is divorced which accounts for 0.24% of the married population, and 0.11% of the total population. More women were separated or divorced than men, and the number of separations was almost three times as high as the number of divorces.

Divorce, a historically taboo topic in India for ages [Dommaraju, 2016], seldom features in mainstream Indian discourse [Goode, 1962]. Recent indications of changing social acceptance of divorcees notwithstanding [Mani and Priya,

∗Ashiqur R. KhudaBukhsh is the corresponding author.

2017], divorce in India still carries a considerable social stigma [Belliappa, 2013].

How do we quantify gender inequality in Indian divorce? Surveys about divorce often have limited participation and a small sample size [Vasudevan et al., 2015], perhaps due to the social stigma attached. A vulnerable community - Indian women under conjugal distress - had limited visibility to social scientists. Via a substantial corpus of 17,306 divorce court proceedings, this paper conducts the first-ever computational analysis of gender inequality in Indian divorce based on public court records.

Even though written in English, legal texts are often domain-specific [Bhattacharya et al., 2019]. The considerable variation of legal jargon across countries and courts makes domain-specific analysis important. In that vein, Indian legal NLP is an emerging field [Bhattacharya et al., 2019; Kalia et al., 2022]. Most NLP research on legal texts thus far has focused on building robust tools to analyze legal text. Recent research, however, on in-group bias [Ash et al., 2021] and sexual harassment [Kumar, 2020], and Figure 1 and Table 1 suggest that automated methods to glean social insights from large-scale, legal texts merit investigation. Barring few recent lines of work [Madaan et al., 2018; Bhattacharya et al., 2020; Khadilkar et al., 2022], there is surprisingly little literature on large-scale linguistic analysis of gender bias in India, let alone on legal text zeroing in on divorce.

While emerging data sources (e.g., public court records available on the web) offer opportunities to social scientists to study important and sensitive social issues that previously had limited survey data, applying cutting-edge NLP methods to newer domains requires careful evaluation of the critical question: How much of the (perceived) gender inequality as quantified by the methods truly reflects the corpus and how much of it is due to the inherent biases of the employed NLP methods? In this paper, we show that the subtleties present in legal text present unique challenges. Unless we consider them and make non-trivial changes to existing methods, we may end up drawing inaccurate social conclusions. We further show that sophisticated NLP methods built on top of large language models (LLMs) need scrutiny when applied to social inference tasks involving genders. We, in fact, conduct a much broader bias audit of these systems. Our audit reveals well-known LLMs often exhibit gender bias even on simple subject-verb-object sentence completion tasks. Through a corpus-specific text entailment analysis, we demonstrate that downstream applications such as natural language inference (NLI) systems also exhibit sensitivity to gender. We finally, present a novel inconsistency sampling method to mitigate this bias and present our social findings.

To summarize, our contributions are the following:

• Social: We create a substantial corpus of 17,306 divorce court proceedings and conduct the first-ever analysis of gender inequality through the lens of divorce proceedings. While a large number of court cases perhaps suggest changing norms in India where women are increasingly challenging patriarchy [Sonawat, 2001], our analyses reveal widespread domestic violence, dowry demands, and torture of the bride.
• Methodological: We address extant gaps and limitations in multiple NLP frameworks. We propose non-trivial modifications to the WEAT framework [Caliskan et al., 2017] to make it suitable for legal text. We demonstrate a novel application of text entailment [MacCartney and Manning, 2008] in quantifying gender inequality. We investigate several potential sources for model bias in NLP resources that can interfere with quantifying gender inequality. We present a novel inconsistency sampling method exploiting counterfactuals to mitigate this bias.

2 Dataset

2.1 Collection

We scrape all the publicly available court proceedings with the word divorce between January 1, 2012 to December 31, 2021 from https://indiankanoon.org/ (hereafter IndK), an Indian law search engine launched in 2008 and the largest free online repository of the court proceedings of different courts and tribunals of India. Prior computational law research [Mandal et al., 2021] and gender focused social science studies [Kumar, 2020] have used IndK as source of data.

We download 86,911 case proceedings containing the word divorce from IndK using its advanced search feature. Filtering based on the keyword divorce is a high-recall approach to obtain relevant cases with precedence in computational social science research [Halterman et al., 2021b; Dutta et al., 2022]. However, presence of the keyword divorce may not always indicate a divorce court proceeding; for instance, the keyword can be used to describe the marital status of any of the litigants. It can also be used in an altogether different context (e.g., divorced from reality). We use the following heuristic to further refine our dataset. We also look for other words (e.g., husband, wife, marriage) and phrases (e.g., decree of divorce), and check if such occurrences repeat for a minimum threshold (set to 5). On a random sample of 100 cases after we apply this cleaning method, a manual inspection reveals that 94 are divorce cases. Hence, our keyword-based filtering is reasonably precise. This pruning step retains 25,635 cases.

2.2 Data Pre-processing

To quantify gender inequality in court proceedings, we must disambiguate the legal parties - the plaintiff and the defendant - and accurately identify of the husband and the wife, who plays which role. Indian legal documents use a wide range of legal terms to denote the plaintiff (e.g., appellant, applicant, complainant, petitioner) and the defendant (e.g., respondent, nonapplicant, opponent). We observe different courts have different formats (sometimes, multiple formats) to summarize the proceedings. The documents also specify which party in marriage represents which role in several different ways (e.g., respondent/wife, respondent-wife, respondent aggrieved wife). We write a regular-expression-based pipeline and consolidate such information to identify the gender of the plaintiff and the defendant across all the states.

The names and salutations (e.g., Mr., Mrs., Smt., Shri) of the plaintiff and defendant also provide gender information. Subcultural naming conventions played a key role in assigning gender to the litigants in some of the cases. For instance, Kaur, meaning princess, is a Punjabi last name only for females [Kaur-Aulja et al., 2019]. Or ben, meaning sister, is solely used in many female names in Gujarat [Mistry, 1982]. Dependence information of the litigants also provides gender information (e.g., son of, daughter of, wife of).

Of the 25,635 cases, we could unambiguously assign gender to 17,306 cases. For each case, we replace each mention of the litigants as wife or husband accordingly. For example, a proceeding snippet "The plaintiff/wife has filed for a divorce. The plaintiff was married to the defendant for three years.", will be modified to "The wife has filed for a divorce. The wife was married to the husband for three years." This data set, Ddivorce, consists of 30,615,754 (30 million) tokens.

3 Brief overview of Indian legal system

Indian Judicial System is largely based on the English Common Law system (where, the law is developed by judges through their decisions, orders, and judgments). The nation has 28 states and 8 union territories (UT), and a total of 25 high courts (some high courts have jurisdiction of more than a state or UT). The federal structure has a supreme court coupled with the high courts that roughly handle the cases in a state or UT. The legal cases of divorce are usually handled by the family or district courts. However, some unresolved cases or sometimes fresh cases are also heard by the high courts. Since the court proceedings are public records and are digitally made available freely by IndK, we found this dataset to be quite appropriate for a large-scale study on gender equality in court proceedings.

4 Methods Overview

We use two NLP methods to quantify gender inequality: (1) Word Embedding Association Test; and (2) a text entailment framework. A brief description follows.

4.1 Word Embedding BasedMethods

The first metric is called Word Embedding Association Test (WEAT) introduced by Caliskan et al. [2017]. To calculate the metric, the words are embedded and the vectors #» a and #» b are obtained for the words a and b respectively. The cosine similarity of these words are denoted by cos( #» a, #» b ). The metric considers two sets of target words given by X and Y, and two sets of attribute words A and B. Then, the WEAT score is defined as WEAT(X , Y, A, B) = (meanx∈X σ(x, A, B) - meany∈Yσ(y,A, B))/stddevw∈X∪Yσ(w,A, B), where, σ(w,A, B) = meana∈A cos( #» w, #» a) - meanb∈B cos( #» w, #» b ).

Intuitively, σ(w,A, B) measures the association of w with the attribute sets, and the WEAT score measures the differential association of the two sets of target words with the attribute sets. A positive WEAT score implies that the target words in X is more associated with the attribute words in A than B and the words in Y is more associated with B than A.

4.2 Text Entailment BasedMethods

Quantifying gender inequality relying on the distributed representation of words presents a diffused, bird's-eye view of the larger trends. Also, these methods are known to be datahungry [Mikolov et al., 2013]. Data availability often becomes a limiting factor to conducting contrastive studies at different spatio-temporal granularity. In what follows, we present a novel application of text entailment, a natural language inference (NLI) task [Dagan et al., 2005] that bypasses the data size requirement and equips us with a finer lens through which we can compare and contrast gender inequality with respect to individual verbs.

An NLI system take a premise P and a hypothesis H as input and outputs entailment, contradiction, or semantic irrelevance. For instance, the hypothesis some men are playing a sport is entailed by the premise a soccer game with multiple males playing [Bowman et al., 2015]. As one can see, textual entailment is more relaxed than pure logical entailment and it can be viewed as a human reading P would infer most likely H is true. This framework has gained traction in several recent social inference tasks that include estimating media stance on policing [Halterman et al., 2021a; Dutta et al., 2022], aggregating social media opinion on election fairness [KhudaBukhsh et al., 2022], and detecting COVID-19 misinformation [Hossain et al., 2020]. Formally, let NLI(P,H) takes a premise P and a hypothesis H as inputs and outputs o ∈ {entailment, contradiction, neutral}. Following [Dutta et al., 2022], we define entailment ratio (denoted by ent(D, H)) for given corpus D and a hypothesis H, as the fraction of the individual sentences present in D that entails H: ent(D, H) = ∑P∈D I(NLI(P,H)=entailment) |D| , where I is the indicator function. A larger value of ent(D,H) indicates greater support for H in the corpus.

Consider we are interested in learning how often the husband and the wife are accused of torture (physical or emotional) in our corpus. We analyze this research question in the following way. We first construct a sub-corpus Dtorture from the divorce court proceedings consisting of sentences that (1) mention husband or wife at least once; and (2) mention torture as a verb at least once. We next construct two hypotheses - HMV,torture and HFV,torture - using a man and a woman as victims and perpetrators interchangeably. HMV,torture is A woman tortures a man and HFV,torture is A man tortures a woman . We next compute the entailment gap defined as

gap(Dtorture, torture) =

ent(Dtorture, HFV,torture) - ent(Dtorture, HMV,torture)

Effectively, this means we compute the fraction of sentences that entail A woman tortures a man in Dtorture and subtract it from the fraction of sentences that entail A man tortures

a woman in Dtorture. An overall positive number indicates that the male has been described as the torturer more often than the female in court proceedings. A negative value would indicate the opposite way. Similar analysis can be extended to other verbs such as assault, beat, or abuse.

5 Design Considerations

Adapting the WEAT and entailment frameworks to quantify gender inequality in our domain requires careful consideration of several aspects described in what follows.

5.1 Verbs for Target Sets

Traditionally, WEAT score is used to quantify gender or racial stereotypes. The majority of the elements present in those attribute sets would be nouns and adjectives (e.g., criminals, terrorists, doctors, police) [Manzini et al., 2019; Greenwald and Pettigrew, 2014] and seldom verbs [Hoyle et al., 2019]. We are interested in understanding the action space of the two parties fighting a divorce case; we want to know if the court described that one party tortured or abused the other. Hence, verbs are a natural choice for our target set.

We inspect the list of high-frequency verbs in the corpus and narrow down to the following ten verbs: Xunpleasant = {abuse, assault, beat, burn, cheat, misbehave, rape, slap, threaten, torture}. A small subset of these words are already present in the list of unpleasant stimuli presented in [Greenwald and Pettigrew, 2014]. We further compute the average valence score of these words as per the lexicon presented in [Warriner et al., 2013]. We find the average valence score of Xunpleasant is 2.7, comparable to the average valence score (2.16) of unpleasant stimuli presented in [Greenwald and Pettigrew, 2014].

Divorce being a bitterly fought family situation, we observe a sparse presence of pleasant verbs such as love, care, or empathy in our corpus. Since infrequent words in the corpus do not have reliable embeddings [Lample et al., 2018], in contrast with traditional applications of WEAT score, we choose the target set Y to be an empty set.

5.2 The Torturer and the Tortured

The attribute sets A and B as defined in the WEAT score represents the identifiers used for the plaintiff and defendant in our data (e.g., A consisting of he, him, husband, and B consisting of she, her, wife etc.). However, notice that WEAT score is agnostic about whether the identifier is the contributor or the receptor of target words. For example, torture does not happen in isolation; it requires a torturer and one who is tortured. Unlike nouns, verbs are typically associated with two entities - the subject and the object. To disambiguate between "the husband tortured the wife " and "the wife tortured the husband ", a word embedding needs to understand this nuance. Otherwise, the embedding is likely to place both the plaintiff and defendant identifiers equidistant to the verb.

To disambiguate these two situations, we run the corpus through the stanza POS tagger [Qi et al., 2020] to find out the subject and object of the sentences and whether the statements are in active or passive voice. Based on this, we classify the subjects and objects as 'male perpetrator', 'female perpetrator', 'male victim', or 'female victim', in the sentences that has the target verbs. We replace these four cases with four unique words (denoted by wMP, wFP, wMV, and wFV, respectively) so that those words do not occur anywhere else in any of the documents. We call this new dataset Dreplaced.

6 Word Embedding Based Analysis

We are interested in two research questions:

RQ 1: How does gender inequality manifest in divorce court proceedings with respect to unpleasant verbs in X?

RQ 2: Is our careful disambiguation of the torturer and the tortured necessary at all?

In order to answer these two questions, we run two sets of experiments with identical training configurations. First, we run experiments on Dreplaced using the target and attribute sets as defined in the previous section. We train the word embedding model 10 times and calculate the WEAT scores for each of the following two cases: when both genders are (a) perpetrators, i.e., when A = {wMP}, B = {wFP}, and (b) victims, i.e., when A = {wMV}, B = {wFV}. We use the default parameters for training our FastText [Bojanowski et al., 2017] Skip-gram embedding with the dimension set to 100 for all word-embeddings in this paper. Second, we run a baseline experiment with the original text data without replacing them with the four unique words (Ddivorce) and use the attribute sets as A = {husband} and B = {wife}. The number of runs and the embedding method are the same in both experiments. The results are shown in Figure 2.

As already described, a negative WEAT score indicates B is more associated with the target set as compared to A. Hence, if we look from the perspective of the victim, we find that women are more associated with the unpleasant verbs than men. In contrast, when viewed from the perpetrator's perspective, a positive WEAT score implies that men are more associated with the unpleasant verbs. Hence, our results indicate that in our corpus, women are more often the victims while men are more often the perpetrators.

Our baseline experiments that do not make any distinction between the perpetrator and the victim give a WEAT score close to zero indicating near-perfect gender equality. This inaccurate result, while highly surprising from a social science perspective, is not unexpected given how the original framework functions. The two entities (husband and wife) are present around the unpleasant verbs with nearly equal frequency. If the method does not make any distinction between the roles of victim and perpetrator, WEAT will give inaccurate results. We thus carefully use the WEAT score to elicit the correct gender bias when applied to legal texts for our social science research question.

7 Societal Inequality andModel Bias

Our word embeddings are computed from scratch while our next set of experiments relies on downstream applications built on top of large language models. Large language models (LLMs) are known to have a wide range of biases due to the train data [Bender et al., 2021] and extant literature has examined gender bias in the form of occupational stereotypes present in NLI systems [Rudinger et al., 2017]. We thus need to disentangle societal inequalities that are potentially reflected in our corpus and model biases that are potentially present in the NLP applications.

Essentially, for a premise/hypothesis pair ⟨P,H⟩, the NLI system estimates the probability P(H |P). However, how LLMs encode the probability P(H) when the hypotheses primarily consists of the two genders (male and female) and a set of verbs is understudied. A thorough investigation first reveals that the masked word prediction probability of several well-known LLMs is sensitive to gender. We next present a measure to quantify gender bias sensitivity of NLI frameworks and present mitigating strategies. Finally, we use a bias-mitigated NLI system on our corpus and report findings.

7.1 Implicit Bias in Agent and Theme in LLMs

Unlike existing literature that primarily target occupational stereotypes to quantify and analyze gender bias [Rudinger et al., 2017; Sharma et al., 2021; Khadilkar et al., 2022; Caliskan et al., 2017; Kumar et al., 2020], we focus on a very basic unit in a sentence - the verbs. Following [Sap et al., 2017], let in a sentence X verbs Y, X represent the agent and Y represent the theme. Many verbs imply the relative authority levels between the agent and the theme. For example, in the sentence The football coach instructed the players to play a conservative game, the agent (the football coach) has more authority than the theme (the players). In contrast, the agent has less authority than the theme in the sentence The football coach honored the players' suggestion to play a conservative game. First proposed in [Sap et al., 2017], the connotation relation of power captures this notion of the power differential between an agent and a theme with respect to a given verb.

While the connotation relation of power has been analyzed in the context of gender inequality in movie scripts [Sap et al., 2017] and follow-on research focused on editorial fixes to remove bias [Ma et al., 2020], little or no literature exists that documents the implicit gender bias present towards the agent and the theme when specific verbs are considered. This research is important and has a broader impact beyond our current social inference task. For instance, if an LLM encodes that it is less likely for a woman to inspire or guide someone than a man, this bias may percolate to downstream tasks leading to erroneous social conclusions when applied to large-scale data for other social inference tasks.

We use cloze tests to evaluate this implicit bias. A brief description of cloze test follows.

• Cloze test: When presented with a sentence (or a sentence stem) with a missing word, a cloze task [Taylor, 1953] is essentially a fill-in-the-blank task. For instance, in the following cloze task: In the [MASK], it snows a lot, winter is a likely completion for the missing word. Word prediction as a test of LLM's language understanding has been explored in [Paperno et al., 2016; Ettinger, 2020].
• Bias Evaluation Framework: We describe our proposed testing framework for gender bias. Let LLMcloze(w, S) denote the completion probability of the word w with a masked cloze task S as input. For a given verb v, we consider the following four cloze tests:
• A [MASK] v a woman (denoted by vwomanAsTheme)
• A [MASK] v a man (denoted by vmanAsTheme)
• Aman v a [MASK] (denoted by vmanAsAgent)
• A woman v a [MASK] (denoted by vwomanAsAgent)

In an ideal world where the LLM treats men and women equally, LLMcloze(man, vwomanAsTheme) and LLMcloze(woman, vmanAsTheme) should be equal. However, our preliminary exploratory analysis indicates that is not the case. For example, when v is set to inspire, BERTcloze(man, vwomanAsTheme) is 0.20 whereas BERTcloze(woman, vmanAsTheme) is 0.16. When we set v to guide, the gap widens - BERTcloze(man, vwomanAsTheme) is 0.71 whereas BERTcloze(woman, vmanAsTheme) is 0.36.

Again, in an ideal world where the LLM treats men and women equally, LLMcloze(man, vwomanAsAgent) and LLMcloze(woman, vmanAsAgent) should be equal.

Let V denote the set of all verbs listed in [Sap et al., 2017] where the agent has more power than the theme. Our overall measures of implicit bias are: (a) (1/|V|) · (∑v∈V (LLMcloze(man, vwomanAsTheme)- LLMcloze(woman, vmanAsTheme))), and (b) (1/|V|) · (∑v∈V (LLMcloze (man, vwomanAsAgent)-

LLMcloze (woman, vwomanAsAgent))).

Measure (a) quantifies biasagent. A positive value indicates that the LLM encodes a man being in the position of agent likelier than a woman on expectation. Measure (b) quantifies biastheme. A positive value indicates that the LLM encodes a man being in the position of theme likelier than a woman on expectation. We investigate three well-known LLMs for this audit: BERT [Devlin et al., 2018]; RoBERTa [Liu et al., 2019]; and Megatron [Shoeybi et al., 2019]. We consider 1,222 verbs listed in [Sap et al., 2017]. We also consider verbs in Xunpleasant for this study.

Table 2 summarizes our gender bias audit of LLMs with respect to verbs implying more power to the agent than the theme. We first note that for both verb sets, biasagent is substantially larger than biastheme. This result indicates that men are considerably more likely to be considered as the agent when women is the theme and the verb implies that the agent has greater power than the theme. We also note that the completions favor mildly men over women even for the theme, however, the values are closer to 0.

7.2 Implicit Bias in NLI Systems

We describe our approach to quantify model bias in our NLI framework specific to our task. Consider we modify the sub-corpus Dtorture to Dflipped torture where the gender identifiers in each premise sentence are flipped to the equivalent identifier of the opposite gender. For instance, the premise The wife tortured the husband both mentally and physically will be modified as The husband tortured the wife both mentally and physically. Flipping gendered words to test bias through counterfactuals in the context of coreference resolution has been previously explored in [Lu et al., 2020]. We argue that if a premise in Dtorture entails A man tortures a woman , the flipped premise in Dflipped torture should entail A woman tortures a man instead in a gender-neutral NLI system. Hence the entailment gap for torture computed on Dtorture should be equal in magnitude and opposite in polarity as the entailment gap computed on Dflipped torture. The NLI system's (M) overall bias score with respect to verbs present in Xunpleasant is thus computed as NLIbias(M,Xunpleasant) = ∑v∈Xunpleasant abs((gap(Dv ,v)+gap(Dflipped v ,v)) |Xunpleasant | . In simple words, for each verb, we compute the entailment gap (value1) for the relevant sub-corpus and the flipped sub-corpus (value2). We subtract value2 from value1 and take the absolute value of the sum. The bias score is the average value of this sum across all verbs: a score close to 0 indicates that the NLI system has a minimal bias, whereas larger values indicate greater bias.

Our baseline is an off-the-shelf NLI system from Allen NLP trained using RoBERTa (denoted by Mbase). We find that NLIbias(Mbase, Xunpleasant) is 0.27 1 .

7.3 Bias Mitigation Via Inconsistency Sampling

Active Learning is a powerful and well-established form of supervised machine learning technique [Settles, 2009] characterized by the interaction between the learner, aka the classifier, and the teacher (oracle or annotator). Each interaction step consists of the learner requesting the teacher the label of an unlabeled instance sampled using a given sampling strategy and augmenting the data set with the newly acquired label. Next, the classifier is retrained on the augmented data set. This sequential label-requesting and re-training process continues until some halting condition is reached (e.g., exceeded annotation budget or the desired classifier performance). At this point, the algorithm outputs a classifier, and the objective for this classifier is to closely approximate the (unknown) target concept in the future. The key goal of active learning is to reach a strong performance at the cost of fewer labels.

1 We note that a bias-aware NLI variant from Allen NLP has a better starting point (bias score 0.20) than the base model. However, the bias-aware model exhibits slower convergence than the base model when we conduct our active learning steps as discussed in Section 7.3. With identical experimental setting, after iteration 3, the bias-aware model improves its bias score to 0.133.

Some of the well-known sampling methods include uncertainty sampling [Settles, 2009], certainty sampling [Sindhwani et al., 2009], and density-based sampling [Nguyen and Smeulders, 2004].

· Inconsistency Sampling: First introduced in Dutta et al. [2022], this sampling technique exploits the underlying logical structure of the ⟨premise, hypothesis⟩ space. For instance, a premise cannot both entail (or contradict) a given hypothesis and its negation. In our work, we extend this idea and exploit a ⟨premise, hypothesis⟩ space richer than Dutta et al. [2022] for logical inconsistency.

Consider the premise/hypothesis pair Continuously her husband used to harass and torture her everyday/A man tortures a woman . We argue that if this premise entails the hypothesis (which it does), the modified premise/hypothesis pair with replacing every gendered word with the opposite gender - i.e., Continuously his wife used to harass and torture him everyday/A woman tortures a man - should also entail. If not, it signals a logical inconsistency. For each sampling iteration, we add 60 samples giving equal weightage to the verbs present in Xunpleasant.

Table 3 summarizes our active learning results. We conduct three rounds of active learning using inconsistency sampling and stop when the performance improvement becomes indiscernible (≤ 0.01). All annotations are independently conducted by two annotators. Since legal documents are typically written in clear, unambiguous language, we observe a near-perfect agreement (Cohen's κ value 0.96). The remaining disagreements are resolved through a post-annotation adjudication step. Table 3 indicates that with subsequent active learning steps, our NLI system exhibits lesser bias. Given that the maximum possible bias score is 2, we achieve substantial improvement in mitigating the bias.

Now that we are more confident that our model inferences are less sensitive to gender, we evaluate the societal bias present in our corpus. Figure 3 summarizes our text entailment results. Barring misbehave, for all other verbs, men are identified as perpetrators more often than women. We further note that verbs that indicate physical abuse, such as rape and beat, particularly stand out with larger values. The average entailment gap for verbs unambiguously indicating physical harm - assault, beat, burn, slap, and rape - is much higher (0.41) than verbs that may or may not indicate physical harm (0.19) such as abuse, cheat, misbehave, threaten, and torture. A manual inspection of randomly sampled 200 ⟨premise, hypothesis⟩ pairs aligns with our automated method's overall findings.

8 Discussions and Limitations

In this paper, we present the first-ever computational analysis (to our knowledge) of gender inequality in divorce court proceedings in India. Based on the documented allegations of parties involved in the divorce, our analyses indicate a striking gender inequality as described in these public records. While documented evidence of marital distress in India exists in social science literature, how such factors play out in divorce has limited understanding. Our study sheds light on a vulnerable and vulnerable and practically invisible community in India.

Methodologically, we identify and address several gaps and limitations of existing NLP techniques to quantify gender inequality. We believe our finding specific to legal text is new, and our method to address it is simple, effective, and intuitive. Casting the problem of quantifying gender inequality as a text entailment task is also new. Our results on text entailment results suggest that NLI can be a viable tool for computational social science researchers to analyze similar research questions (e.g., who gets the child custody can be estimated with hypotheses the husband gets the custody of the child and the wife gets the custody of the child). Moreover, our bias mitigation strategy exploiting a novel inconsistency sampling technique using counterfactuals holds promise.

Our work has the following limitations.

• Sentence level processing: An important point to keep in mind is that our analyses operate at the sentence level. If in a court proceeding, a sentence records that the plaintiff accuses the defendant of wrongdoing which the defendant denies in

a subsequent sentence, how these two contradicting claims are resolved in the court cannot be inferred without language models that can handle document-level contexts. We believe our research will open the gates for investigation with newerage LLMs that can handle broader contexts.

• Archival limitation: The sparse presence of the NorthEastern region in our dataset is most likely due to archival limitation as some of these states record the highest rate of divorce [Jacob and Chattopadhyay, 2016]. Our study is also limited by the overall archival extent of IndK.
• Economic independence: Some of the court proceedings mention the litigants' occupations. We annotated randomly 100 sampled occupations for women. While an overwhelming majority of the sampled occupations are homemakers, compared to World Bank Data of labor force participation of women in India (23%), 32% of the women are working women in our sampled occupations. Economic independence and divorce merits a deeper exploration.
• Out of court settlements, separation, abandonment: Finally, not all unhappy marriages end up in divorce and reach court for the dissolution. Many out of court settlements happen. As documented in Jacob and Chattopadhyay, [2016], the number of separated women in 2011 is almost three times as the number of divorced women. Since divorce is still looked at as a social stigma [Belliappa, 2013] and family institutions are highly valued in India, there could be many women who continue with their dysfunctional marriage while unhappy. The court does not know their stories.

Ethical Statement

We work with public court records. Prior studies exist on Indian court proceedings [Ash et al., 2021]. We conduct aggregate analysis refraining from presenting any personally identifiable information in the paper. Hence, we do not see any ethical concern. Rather, we believe our findings and methods can be valuable to policymakers and social scientists.

A study on binary gender inequality runs the risk of oversimplifying gender, which we acknowledge lies on a spectrum. Same-sex marriage is yet not legal in India. Further nuances will be needed to extend our work to other cultures allowing same-sex marriages. We are also sensitive to previous studies that point out the potential harms of the erasure of gender and sexual minorities [Dev et al., 2021].