For Women, Life, Freedom: A Participatory AI-Based Social Web Analysis of a Watershed Moment in Iran's Gender Struggles

1 Introduction

Words are the only victors.

- Salman Rushdie; Victory City; 2023.

On 16 September 2022, Mahsa Amini, a 22-year-old woman died under police custody in Iran. Reportedly, she was arrested because of not wearing her hijab (headscarf) properly. As media and police presented conflicting accounts of her death [Alkhaldi and Mostaghim, 2022], Mahsa Amini's death enraged Persian (Farsi) Twitter users in an unprece#مهسا امینی dented manner [Kermani, 2023]. The hashtag (#MahsaAmini) became one of the most repeated hashtags on Persian Twitter and initiated a Twitter protest where Iranians expressed their grievances against the government like never before. Support and solidarity for gender equality poured over in from prominent world leaders [France-Presse, 2022], artists [Pina, 2022], and sports personalities [Alkhaldi, 2022] across the globe.

∗Ashiqur R. KhudaBukhsh is the corresponding author.

#MahsaAmini was undoubtedly the overwhelming toptrending hashtag on Persian Twitter for months during the relentless protest. However, for a brief period of time, hashtags with an opposite stance toward the protest (e.g., #ExecuteThem or #ISupportKhamenei 1 ) trended. Prior literature conjectured state-aligned trolling in Iran on Instagram [Kargar and Rauchfleisch, 2019]. Also, social bot accounts' capability to spread extreme ideology is well-documented [Stella et al., 2018; Berger and Morgan, 2015].

Via a substantial corpus of 30.5 million tweets relevant to the protest, this paper makes three key observations:

1. The grievances of protesters against the current government mention a broad range of incidents spanning decades.
2. With respect to account creation time, between the statealigned Twitter accounts and pro-protest Twitter accounts, pro-protest accounts are more similar to baseline Persian Twitter activity.
3. There was a noticeable shift in positive stance toward gender equality after the protests on Persian Twitter discourse.

To our knowledge, no computational analysis relying on sophisticated natural language processing methods exists that has examined gender equality in Persian social media discourse let alone at this unprecedented scale. That said, we believe our key contribution lies elsewhere. Our paper marks an important effort to include the stakeholders - the Iranian women - in this AI-building process. All examples in our supervised solution's training set are annotated by Iranian women. Our examples are thus grounded in cultural contexts and first-person experience about the gender struggles in Iran.

Datasets addressing issues faced by vulnerable communities often end up being annotated by annotators with little or no documentation [Guest et al., 2021; Ramesh et al., 2022]. Since annotated examples often form the core of a supervised AI system, it is important to involve stakeholders in the annotation process. For example, Ramesh et al. [2022] present a lexicon of queer-related inappropriate words where one of the annotators identifies as queer. Similarly, Guest et al. [2021] present a misogyny dataset where the majority of the annotators identify as women.

1 Ali Khamenei is the second and current supreme leader of Iran who is in office since 1989.

Our annotators' role is not limited to mere annotation. Rather, they take an active role in guiding how to curate more meaningful data by suggesting suitable keywords to curate our dataset and providing a valuable seed set of examples to initiate an active learning pipeline. Our results indicate that the annotators' contributions yielded a richer seed set than a random baseline.

At a philosophical level, we see this work as a part of the growing conversation of participatory AI [Harrington et al., 2019; Delgado et al., 2022; Bondi et al., 2021; Birhane et al., 2022] where the goal is to develop systems for the people and by the people.

2 Datasets

As we already mention, #MahsaAmini initiated a protest with global participation. Understandably, tweets in global languages such as English or French are likelier to reflect the global perspective on this issue. Given that Twitter is banned in Iran and users reportedly use VPNs to access Twitter [Kermani, 2023], considering geo-tagged tweets is not a reliable option either to understand and analyze the Iranian perspective. Therefore, we restrict our analyses to only tweets authored in the Persian language. We assume that our choice of language can act as an effective filter to ensure our dataset is less likely to be diluted by the global discourse. We use Twitter's official language label as ground truth.

We collect three corpora: Dprotest ; Dgender ; and Dbaseline . Our dataset spans the time duration of Jan 15, 2022, to Jan 15, 2023 2 . We define the time period from January 15, 2022, to September 15, 2022, as Tbefore. We define the time period from September 16, 2022, to January 15, 2023, as Tafter. A short description follows next. Throughout the paper, if we use a Persian word or phrase, we present an English translation in parentheses following the word.

2.1 Dgender

Dgender consists of 6,036,012 Tweets which has been posted by 700,189 unique users.

• "دختر" or (women) "زنان" 1. All tweets that have either (girl).
• im- the means which "ناموس" 2. All tweets that have either mediate female family members (daughter, mother, sister, wife) whom the male member of the family (father, brother, husband) should protect and sometimes control jealousy of form positive the means which "غیرت" or that men have upon their female family members against other men. These two search keywords were suggested by our annotators.
• All tweets that have gender insult words against women "a or prostitute" "a indicating both "کصده" and "جنده" promiscuous woman" in a pejorative way (the second insult word mostly accompanies with sister).

2 On January 7, 2023, two executions relevant to this protest happened [Radford and Fowler, 2023]. We thus set our end date one week after the executions.

1. all tweets that have at least one word from the two fol"خواهر" (woman), "زن" (girl), "دختر{" lowing subsets: sister}; and
2. (rights), "حقوق" ,)noitulover("انقلاب" (life), "زندگی{" (freedom)}. "آزادی"

2.2 Dprotest

Dprotest consists of:

• yield- them in (#MahsaAmini) #مهسا امینی 1. tweets with ing 21,308,449 Tweets posted by 655,303 unique users.
• tweets that support government through the hashused been has hashtag This .#لبیک یا خامنه ای tag 1,051,792 times by 71,484 unique users across the entire Twitter timeline accessible through the APIs.
• means which #اعدام کنید 3. tweets that have the hashtag execute them. This hashtag has been used 11,292 times by 5,130 unique users across the entire Twitter timeline accessible through the APIs.

2.3 Dbaseline

In order to estimate baseline Persian Twitter behavior, we col- and )که ,از ,در ,با ,به( consider five Persian stop words lect 6,000 tweets per day (evenly distributed across the hours) that contain at least one of these stop words. Our dataset, Dbaseline , consists of 2,190,000 tweets.

We compute the unigram distributions of subsets of Dbaseline that was authored during Tbefore and Tafter . Table 1 lists the top 20 high-frequency non-stop words present when (1) we subtract the unigram distribution of Tafter from the unigram distribution of Tbefore (left); and (2) we subtract the unigram distribution of Tbefore from the unigram distribution of Tafter (left). In plain English, these are the words that appeared more frequently during one period and much less frequently during the other. From the right column of Table 1, we note that several of these words are not indicative of civic unrest while the left column does not indicate similar unrest. We conduct a similar experiment to track shift in high-frequency hashtag usage between the two time periods. We again observe that even in the baseline Persian Twitter discourse, Tafter showed several hashtags relevant to the protest.

3 Baraye - Because Of

A large fraction of tweets of Dprotest contains a phrase (because of ). These tweets were an outlet for Persian Twitter users to vent their frustrations about the situation. Specifically, the tweets aimed at answering a reason for the protests. The tweets expressing a complex collection of emotions on why the current government has failed the nation captured the imagination of Shervin Hajipour, a talented Iranian singer who won the 2023 Grammy award for his song Baraye (because of ). Each line of this song starts with because of and paints a picture of Iranian hope and despair. We collect 1.92 million tweets from Dprotest with the phrase because of. We train a FastText [Bojanowski et al., 2017] word embedding on Dprotest and for each tweet, we compute the line in the Baraye song that is the nearest neighbor in the embedding space. Table 3 lists the top 10 lines from the Baraye song that matched with the tweets.

While the poignant song by Hajipour brilliantly captures Iranian aspirations and struggles, Table 4 indicates there is much more to Iranian angst than what the song could hold. The second most common trigram indicates the access barrier to Twitter. Most prominent social media platforms are blocked in Iran [Kargar and McManamen, 2018; Sohrabi, 2021] and reportedly, Iranians primarily take recourse to VPNs to participate in the social web [Kermani,

2023]. We observe that Navid Afkari 3 , an executed Iranian wrestler, is mentioned among the top trigrams. Multiple structural failures got also mentioned in the common trigrams [Bozorgmehr, 2017]. From the Cinema Rex fire incident that happened in 1978 [Ali and Ali, 2018], to the attacks on student-dormitory in 1999 [Bozorgmehr, 2017], to the current reality of web censorship, the most striking takeaway from Table 4 is perhaps the time duration between the events people mentioned.

4 Account Creation Time

Prior literature has examined state-aligned trolling in Iran on Instagram platforms [Kargar and Rauchfleisch, 2019]. In this section, we present an analysis based on account creation time. We define four sets of users: Upro-protest; Ustate-aligned; Upro-execution; and Ubaseline. Upro-protest represents all users who used the hashtag #MahsaAmini (indicating support for Mahsa Amini) at least once in our dataset. Ustate-aligned represents all users who used the hashtag #ISupportKhamenei (indicating support for Ali Khamenei) at least once in our dataset. Upro-execution represents the set of users who used the hashtag #ExecuteThem at least once in our dataset. Finally, Ubaseline is the set of unique users who contributed to Dbaseline. We compute the account creation time for each user at the granularity of months and obtain normalized histograms for each of these sets. Figure 1 illustrates the account creation temporal distributions of the four user sets. All subsets exhibit a sharp spike around September 2022, however Upro-execution exhibits two different spikes. In fact, in September 2022, Google Trends indicates one of the most popular search queries from Iran Twitter). (download "دانلود توییتر" was

Table 5 computes the KL divergence of the account creation time distributions with respect to Ubaseline. We observe that the distribution of Upro-protest is closest to Ubaseline while Ubaseline is the farthest. The order remains unchanged with other distributional distance measures (e.g., Bhattacharyya distance). Our qualitative findings remain unchanged even if we limit Ustate-aligned and Upro-execution to only those user accounts that used these hashtags during Tbefore and Tafter.

3 https://www.hrw.org/news/2020/09/12/iran-suddenlyexecutes-wrestler-navid-afkari

5 Annotation

All our annotation was conducted by four different annotators. All annotators identify as Iranian women and are fluent speakers of Persian. All of them have undergraduate degrees.

Annotation task. Our text prediction task is to predict the stance toward gender equality. For each tweet, we ask the annotator: does this short document indicate a positive, neutral, or negative stance toward gender equality?

Inter-rater Agreement. Between any two annotators, we have at least 500 overlapping samples. Across all rounds of annotation, the Cohen's κ ranged from 0.41 to 0.52. On a misogyny annotation task, Guest et al. [2021] reported Fleiss' κ of 0.48 and the Krippendorf's alpha as 0.49. Sanguinetti et al. [2018] report category-wise κ = 0.37 for offence and κ = 0.54 for hate. We further note that our observed interrater agreement is higher than Gomez et al. [2020] (κ = 0.15) and Fortuna and Nunes [2018] (κ = 0.17).

Disagreement resolution. Since our task is likely to be subjective, resolving disagreements has to be grounded in the literature. Prior literature has considered diverse approaches to resolving inter-annotator disagreements (e.g., majority voting [Davidson et al., 2017; Wiegand et al., 2019] or third objective instance [Gao and Huang, 2017]). We resolve any disagreement in the following manner. For positives and neutrals, we only consider consensus labels. Following Golbeck et al. [2017], if any annotator marks an example as negative and the other annotator marks it as negative or neutral, we consider the aggregate label as negative. In order to ensure the anonymity of the annotators, we do not conduct any postannotation adjudication step to resolve disagreements.

5.1 Toward Participatory AI

A notable feature in our work is the active involvement of Iranian annotators in both corpus creation and annotation. Our annotators helped us in the following two ways.

Search keywords. At a deeper level, which data could contain relevant information may require a clear understanding of the social realities. While constructing Dgender , choosing woman or girl and gendered insults as search keywords required little cultural context. However, our annotators sugbe to "غیرت" and "ناموس" gested nuanced keywords such as "ناموس" included in our list of search keywords. Recall that, means the immediate female family members (e.g., daughter, mother, sister, or wife) whom the male members (e.g., father, brother, or husband) should protect and sometimes control; have men that jealousy of form positive a means "غیرت" and upon their female family members against other men.

Seed set. A notable feature of our work is the active involvement of Iranian women in the annotation process, where they not only provide labels but also present important representative short documents to construct meaningful seed sets during the guided sampling step described in Section 6.

6 Active Learning Pipeline

Research question: Is there a noticeable change in support for gender equality in Persian Twitter discourse before and after the demise of Mahsa Amini while in police custody?

To estimate the support for gender equality in Persian Twitter discourse, we build a robust classifier detecting content supportive of gender equality. Since hashtag hijacking [Hadgu et al., 2013] is a common phenomenon where users with opposite views may use the most-popular hashtag to express an opposite stance, our goal is to predict the stance toward gender equality from tweet texts only.

We first estimate to which extent tweets supporting gender equality are present in Dbaseline. We randomly sample 500 tweets weighing both Tbefore and Tafter equally (i.e., 250 from each time slice). In addition, we randomly sample 1,000 tweets from Dgender weighing equally Tbefore and Tafter . Table 6 summarizes the label distribution. We note that a large fraction of Dbaseline consists of neutral tweets.

In order to construct a dataset that is diverse and representative of the unlabeled pool, we present an active learning pipeline that consists of well-known sampling steps. A short description of active learning follows next.

6.1 Background

Active Learning is a powerful and well-established form of supervised machine learning technique [Settles, 2009]. It is characterized by the interaction between the learner, aka the classifier, and the teacher (oracle or labeler or annotator) during the learning process. At each iteration, the learner employs a sampling strategy to select an unlabeled sample (unlabeled samples) and requests the supervisor to label it (them) in agreement with the target concept. The data set is augmented with the newly acquired label, and the classifier is retrained on the augmented data set. The sequential labelrequesting and re-training process continues until some halting condition is reached (e.g., annotation budget is expended or the classifier has reached some target performance). At this point, the algorithm outputs a classifier, and the objective for this classifier is to closely approximate the (unknown) target concept in the future. The key goal of active learning is to reach a strong performance at the cost of fewer labels. Since retraining the model and running inference on a large, unlabeled pool is computationally costly, prior literature has examined the trade-offs present in a batch active learning setting [Yang and Carbonell, 2013]. In this work, we follow the batch active learning setting.

6.2 Seed Set Construction

Random Sampling. In order to capture a diverse set of examples, we randomly select 1,000 samples fromDgender and 500 samples from Dbaseline. Table 6 indicates that solely relying on Dbaseline to construct the seed set will result in extreme class imbalance with very few positives and negatives and predominantly neutrals. Sampling from Dgender might yield slightly more positives (and negatives), however, a keyword-based starting point runs the risk of biasing the whole active learning pipeline. In what follows, we present a guided sampling approach similar to Palakodety et al. [2020].

Guided Sampling. When faced with the challenge to find high-quality positive examples championing the Rohingya community, Palakodety et al. [2020] proposed a documentembedding-based, guided sampling method where annotators provide example short documents conforming to a given label. We employ a similar technique where we asked three annotators to provide five examples each indicating positive and negative stances toward gender equality. For each example, we select 25 unique nearest neighbors in the document embedding space from the unlabeled pool giving equal weightage to tweets from Tbefore and Tafter . This yields 750 samples. Upon annotation and resolving disagreements, we obtain 166 positives, 145 negatives, and 231 neutrals. We note that our sampling method yielded substantially more positives (and negatives) than the random sampling baseline.

Table 7 presents a few randomly selected positive and negative seed examples provided by our annotators. We observe that the examples are grounded in women's cultural struggle in Iran [Kazemzadeh, 2002]. Beyond discussions around hijab, inequality in marital and inheritance law [Doherty et al., 2021], restrictions on activities such as visiting stadiums to watch football [Lewis, 2019; Abtahi et al., 2022] echoed in these examples.

Table 8 lists a random sample of retrieved tweet texts when we used the guided sampling method. This table shows that not only we found more positives (and negatives) than the random baseline, but the tweet texts also exhibit richness, diversity, and nuance.

Overall, we obtain 343 positives, 440 negatives, and 1,051 neutrals from the random sampling and guided sampling step. In what follows, we describe two well-known sampling strategies that we employ to further expand our dataset.

6.3 Certainty and Uncertainty Sampling

Certainty sampling. Since our goal is to use the trained model for a social inference task, it is important to rectify high-confidence misclassifications. Minority class certainty sampling has found its use in rectifying high-confidence misclassifications involving short documents such as movie reviews and messages [Sindhwani et al., 2009; Attenberg et al., 2010]; search queries [KhudaBukhsh et al., 2015]; and comments on YouTube videos [Palakodety et al., 2020; Yoo and KhudaBukhsh, 2023]. We conduct certainty sampling for the positive class and select 750 instances that the model predicts as positive with the highest confidence. We also conduct certainty sampling for the negative class and select 750 instances that the model predicts as negative with the highest confidence. In this step, we obtain 338 positives, 345 negative, and 487 neutrals.

Uncertainty sampling. Uncertainty sampling is one of the most well-known sampling strategies used in active learning [Settles, 2009]. Since we have multiple label categories in our prediction task, we use margin sampling, an active learning variant designed for multiple labels [Scheffer et al., 2001]. In this step, we sample 1,500 examples. Upon annotation and resolving the disagreements, we obtain 115 positives, 247 negatives, and 819 neutrals.

To summarize, our active learning pipeline consists of the following steps:

1. Construct an initial seed set by randomly sampling from Drandom, and Dgender, and using guided sampling (Dseed : 343 positives, 440 negatives, and 1,051 neutral instances) using random sampling.
2. Conduct certainty sampling on the positive class and certainty sampling on the negative class (Dcertainty : 338 positives, 345 negatives, and 487 neutral instances).
3. Finally, conduct uncertainty sampling (margin sampling) (Duncertainty : 115 positive, 247 negative, and 819 neutral instances).

Overall, we obtain 796 positive, 1,032 negative, and 2,357 neutral examples.

6.4 Model Performance and Analysis

Table 9 summarizes the performance of our trained models. The performance improves at each active learning step and we finally achieve a Macro F1 performance of 73.27%. We note that if we train a binary classifier with just the positive class and the negatives and neutrals clubbed together as the notPositive class, it is possible to achieve slightly better performance (Macro F1: 77.76 ± 2.29).

To track shifts in stance toward gender equality, we run inference using Mcertainty on Dbaseline. Figure 2 indicates that the discourse became more polarized during Tafter with both percentages of tweets expressing positive and negative stances increasing. However, we also observe that the increase in positive discourse (by a factor of 2.89) is greater than the increase in negative discourse (by a factor of 1.50).

7 Discussions

In this paper, we present the first-ever computational analysis (to the best of our knowledge) of the stance toward gender equality in Persian Twitter discourse following a watershed moment in Iran's history. Our analyses reveal that the grievances of Persian Twitter users against the government span decades and the protest following Mahsa Amini's death perhaps presented an outlet for the angst harbored for a long time. Second, we observe that the distribution of account creation time can present important signals. We find that with respect to account creation time, pro-execution and statealigned user sets are distributionally different from baseline Persian Twitter users.

We follow an ensemble active learning pipeline to construct a robust classifier that detects stance toward gender equality. As a step towards participatory AI, our annotators take an active role in building our machine learning model. There is a growing concern that our ML conversations barely include marginalized community which can further widen the gap of AI-haves and AI-have-nots. All our annotators are Iranian women, with first-person experience of gender struggles. Their role in our system was far more profound than typical annotators. In a guided sampling step, they provided seed examples to expand our dataset lending cultural grounding. They also suggest important keywords to curate our dataset.

Table 4 suggests that a major Iranian grievance is limited access to the internet. Free and fair access to Twitter was many users' wish. While we were working on this paper, Twitter as a platform underwent several significant changes. With the deprecation of academic Twitter and developer accounts being monetized, at this point, it is unclear howmuch of our collected data would be accessible in the future and at what cost. This is a curious juxtaposition of a community longing for access to a platform to voice their concerns while the very same platform is limiting academic researchers' access to study global politics.

On top of the current uncertainties surrounding Twitter, the inherently transient nature of the social web, censorship, and fear of persecution can contribute to missing content for posthoc analyses. In that sense, our paper is a humble attempt to preserve a vulnerable chunk of the social web that chronicled a watershed moment in the gender struggles of Iranian history.

Ethical Statement

We use publicly available tweets collected using academic Twitter API. Since our data is highly sensitive, we only conduct aggregate analyses without revealing personally identifiable information. We also do not conduct any post-annotation adjudication steps that are typical to many annotation tasks to ensure the privacy of the annotators.

We trained our model on top of a large language model. Several lines of recent research have indicated that large language models have a wide range of biases that reflect the texts on which they were originally trained, and which may percolate to downstream tasks [Bender et al., 2021].

Acknowledgements

We thank Lipika Mazumdar and Shawnee Fereydouni for their input.