Harnessing Code Switching to Transcend the Linguistic Barrier

1 Introduction

Analyzing geopolitical events through the lens of social media is a highly active research domain. From referendums with far-reaching political consequences (e.g., Brexit [Celli et al., 2016]) to sensitive and highly polarizing issues like mass shootings in the US [Demszky et al., 2019], large scale social media analysis has the potential to offer important insights to political and social scientists. While social media discussions lend a great platform to exchange ideas, share opinions and debate issues, tackling online attacks targeted at certain individuals, or communities forms an important modern-day social media challenge to ensure human well-being.

∗Ashiqur R. KhudaBukhsh and Shriphani Palakodety are equalcontribution first authors. Ashiqur R. KhudaBukhsh is the corresponding author.

†Deceased 28 February, 2020.

Typical approach to moderate online hate consists of detecting hate speech for subsequent moderation. However, one recent line of work argued in favor of identifying positive content in the context of heated online discussions between nuclear adversaries at the brink of waging full-fledged war [Palakodety et al., 2020a]. In a substantial corpus of 2.04 million YouTube comments on videos relevant to the 2019 India-Pakistan conflict triggered by the Pulwama terror attack, Palakodety et al. [2020a] advocate the importance of hostility-diffusing comments and define a new task of detecting hostility-diffusing hope speech.

While Palakodety et al. [2020a] present an important study of modern conflict between two nuclear adversaries with a long history of acrimonious past and grim projection of consequences should there be a full-blown war (100 million projected deaths as forecast by Toon et al. [2019]), typical to several studies conducted in linguistically diverse regions, the focus is largely restricted to the English subset of the comments (921,235 English comments posted by 392,460 users). With a combined language base of more than 500 million speakers of Hindi in India and Pakistan as opposed to nearly 250 million English speakers, and a considerable fraction using Romanized Hindi on the web [Gella et al., 2014; Palakodety et al., 2020a], extending this result to the Romanized Hindi subset of comments has understandable benefits. However, such omissions are common in social analyses due to lack of NLP (Natural Language Processing) resources in the native language. Most NLP pipelines are monolingual - state of the art POS (Part-of-Speech) taggers, parsers, or NER (Named Entity Recognition) taggers are rarely trained to handle multi-lingual documents and largely focus on English. Moreover, apart from typical colloquial style social media content, the presence of code mixing [Gumperz, 1982; Myers-Scotton, 1993] - seamless alteration between multiple languages within the same document boundary (e.g., a tweet, or a comment on a YouTube video) - makes the task substantially more challenging.

While the challenges posed by code switching to downstream analyses are well-documented, in this paper, we focus on a largely under-explored research question: How can code switching be harnessed for social good and human wellbeing?

Our research question is motivated by a simple intuition that a short text document is likely to express a consistent sentiment; if reliable linguistic separation of such code mixed documents can be achieved, the Hindi portion of the comments can be further harnessed to explore similar comments in the Hindi subset for which we require no further training (the hope speech classifier we used in this paper is trained on English comments). Effectively, our method uses the Hindi portions of code mixed comments as a seed set to mine similar content authored in Hindi. Our approach presents a compelling case study on how code switching can be harnessed to perform cross-lingual sampling and detect peace-seeking content written in a low-resource language. A reliable system to identify code mixed hope speech documents has additional untapped benefits. Intuitively, a code mixed document written in two dominant languages in a linguistically diverse region is likely to be partially accessible to a wider set of audience.

Contributions. Our contributions are the following.

1. Human well-being: We focus on the important task of detecting hostility-diffusing hope speech [Palakodety et al., 2020a]. Social media is poised to play an increasingly important role in understanding and analyzing modern conflicts [Zeitzoff, 2017]; online discussions between countries with a long history of conflicts are under-studied yet highly important. Additionally, we demonstrate our method's generalizability through a new task introduced in this paper - detecting comments authored in a low-resource language encouraging compliance with COVID-19 health guidance.
2. Framework: Code switching is typically viewed as an impediment to effective corpus analysis; to the best of our knowledge, our work is the first to highlight its untapped potential as a bridge between sub-corpora authored in different languages to perform cross-lingual sampling. While the role of mother tongue as a conversational lubricant in a code switched environment has been previously studied in educational settings [Butzkamm, 1998], harnessing code switching to effectively sample content from a sub-corpus written in a different language has, to our knowledge, never been explored before.
3. Machine Learning: We leverage recent literature in language identification and Active Sampling to sample documents exhibiting high levels of code mixing and provide an end-to-end pipeline to sample from Romanized Hindi starting with a hope speech classifier trained on English documents. Our results indicate that our approach considerably reduces manual effort in acquiring hope speech written mostly in Romanized Hindi.

Organization of the paper. The rest of the paper is organized as follows. We present literature relevant to our research in Section 2, our problem definition, pipeline and necessary background in Section 3, a detailed description of our methods and performance on hope speech detection in Section 4, and an exploratory study on the COVID-19 crisis [Johns Hopkins, 2020] in Section 5. We finally end with our conclusions and proposed extensions in Section 6.

2 RelatedWork

Code switching has been a widely studied area in linguistics for nearly half a century [Auer, 2013]. While recent work on analyzing the social aspects of code mixing in online communities is gaining importance [Yoder et al., 2017], typically, code switching is viewed as an impediment to downstream NLP analyses and much of the focus in the community is concentrated in token-level language identification and switch point detection for cleaner linguistic separation [Das and Gamb¨ack, 2014; Rijhwani et al., 2017a; Gella et al., 2014]. To the best of our knowledge, harnessing code switching for social good and human well-being has been largely unexplored. Our work draws inspiration from field-work in classroom settings showing how code switching helps students overcome linguistic barriers and how native tongue is used in a code mixed setting as a conversational lubricant [Butzkamm, 1998].

Our work focuses on an important domain of online hostility-diffusion between civilians of nuclear adversaries [Palakodety et al., 2020a]. We use several resources presented in the paper (e.g., data set, language identification method with minor modification). However, in the work by Palakodety et al. [2020a], the primary focus was mostly restricted to the English subset of comments, whereas in our work, we focus on leveraging an untapped potential of code switching and propose a pipeline to identify hostilitydiffusing hope speech from the Hindi sub-corpus, a task previously not addressed.

In Gella et al. [2014], the importance of a robust tokenlevel language identification systemwas explored. The study demonstrated that typical document-level language identification systems are a poor fit for code mixed documents. In the context of Indian social media, Gella et al. [2014] also provided statistics on the use of Romanized Hindi, and code mixed text revealing significant use. Language preferences to express opinion were further investigated by Rudra et al. [2016] revealing that negative opinion is often presented in Hindi. The utility of code switching in improved success rates of Wikipedia edits was studied in [Yoder et al., 2017]. Several studies have addressed challenges in analyzing code mixed text by using a token-level language-identification step in their NLP pipelines [Nguyen and Dogruoz, 2013; Elfardy and Diab, 2013; Rijhwani et al., 2017b]. Rijhwani et al. [2017b] in particular presented an HMM-based unsupervised token-level language-identification method to analyze code-switching statistics on social media.

Recent studies have used sentence embeddings for sampling comments similar to a "query" document [Dimovski et al., 2018; Kumar et al., 2019; Palakodety et al., 2020c]. We utilize the polyglot embeddings themselves as sentence embeddings in our nearest-neighbor sampling method.

3 ProblemDefinition and Background

3.1 Low-resource Language

Low-resource or under-resourced languages lack computational resources such as large corpora (monolingual or parallel) and annotated resources typically needed for NLP methods (e.g., parsers, Named Entity Recognition taggers etc.) [Cieri et al., 2016]. Romanized Hindi or Bengali are two examples of highly prevalent yet low-resource languages.

3.2 Task: Hope Speech Detection

We focus on the prediction task of hope speech detection in the context of online discussions relevant to the 2019 IndiaPakistan conflict [Palakodety et al., 2020a]. Aimed at diffusing hostility, a hope speech classifier is a nuanced classifier to detect content that contains a unifying message focusing on the war's futility, the importance of peace, and the human and economic costs involved, or expresses criticism of either the author's own nation's entities or policies, or the actions or entities of the two involved countries (for precise definition with illustrative examples, see [Palakodety et al., 2020a]).

Data set. Our data set, D, consists of 2.04 million comments posted by 791,289 user on 2,890 YouTube videos relevant to this India-Pakistan conflict. Our main focus is on the English and Romanized Hindi subsets denoted as Den (921,235 comments) and Dhe (1,033,908 comments), respectively. For the remainder of this paper, we use Romanized Hindi and Hindi interchangeably.

Annotated data set. The hope speech classifier is trained on an annotated data set, Dtrain hope , of 2,277 positive and 7,716 negative English comments and an in-the-wild performance (on data not belonging to the training or test set) of 84.68% precision was reported.

3.3 An Illustrative Example

To motivate our intuition, we first provide an illustrative example of a code switched comment exhibiting hope speech along with a loose translation. English, Hindi and neutral tokens (e.g., proper nouns, numerals, or technology terms) are color-coded with blue, red and black respectively (color scheme is consistent throughout the paper).

I am Indian and I say peace is the only solution ankh k badle ankh mangoge toh sari dunya andhi hojayegi

I am Indian, and I say peace is the only solution; an eye for an eye makes the whole world blind.

In the above example, both the Hindi and English components exhibit peace-seeking intent. Our main goal in this paper is to harness the Hindi components present in these highly code mixed hope speech comments to detect hope speech in the Hindi sub-corpus. Associated research questions are the following:

• How can we sample code mixed documents?
• How can we harness the Hindi part of a code mixed document to sample hope speech from the Hindi portion?

3.4 AChallenging Data Set

Similar to most data sets of noisy, short social media texts generated in a linguistically diverse region, our data set exhibits a considerable presence of out-of-vocabulary (OOV) words, code mixing, and grammar and spelling disfluencies. In addition to these challenges, given that a vast majority of the content contributors do not speak English as their first language, we noticed varying levels of English proficiency in the corpus with a substantial incidence of phonetic spelling errors (e.g., [thankyou pakusta for hiumaniti no war aman ssnti kayam kare] loosely translates to Thank you Pakistan for humanity; let peace prevail.); 32% of times, the word liar was misspelled as lier. Since Romanized Hindi does not have any standard spelling (e.g., the word aman meaning peace is spelled in the corpus as amun, amaan and aman), a high level of spelling variations added to the challenges.

How hard is it to sample hope speech? On a random sample of 1,000 comments from Dhe , our annotators 1 found 18 positives (i.e., 1.8%). This result aligns with results reported by Palakodety et al. [2020a] where only 2.45% randomly sampled English comments were marked as hope speech. Additionally, a previous study of a multilingual Hindi-English tweet corpus observed that Hindi was more commonly used to express negative sentiment [Rudra et al., 2016]. The minuscule presence of hope speech indicates that detecting such content is essentially a rare positive mining task and automated methods are essential.

3.5 Our Pipeline

Research question: How to harness code switching to sample hope speech from the Hindi subset Dhe ?

A schematic diagram of our pipeline to sample hope speech from the Hindi subset, Dhe , is presented in Figure 1. Our pipeline consists of the following steps.

1. Identify the subset, Dcm, fromDhe ∪Den with substantial code mixing.
2. Run the hope speech classifier (trained on annotated English comments Dtrain hope ) on Dcm and construct the subset Dhope containing comments predicted as hope speech.
3. Construct Dhope he transforming each comment in Dhope discarding any tokens not written in Romanized Hindi.
4. Using Dhope he as the seed set, retrieve the nearest neighbors in the comment embedding space from Dhe .
5. Manually inspect the obtained sampled comments to detect hope speech.

Steps 1, 2, 3, and 4 require minimal manual supervision. Step 5 is the only step that requires substantial manual effort. Our results indicate that we obtained a nearly 10-fold improvement over our baseline (random sampling yields 1.8% hope speech).

1 For all tasks, two annotators proficient in English, Hindi, and Urdu were used. Across all rounds of labeling, the minimum Fleiss' κ measure was high (0.84) indicating strong inter-rater agreement. After independent labeling, differences were resolved through discussion.

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

n

k

5

R

Z

m

u

z

V

3

S

c

W

w

L

f

U

>

A

C

X

D

N

F

r

P

d

O

G

y

q

I

Q

Y

B

/

j

T

p

M

K

0

+

7

8

v

2

H

g

E

9

o

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

C

R

0

f

o

7

d

P

q

K

+

Z

8

c

r

n

I

>

A

B

X

V

D

S

N

F

2

p

U

u

v

g

G

y

m

k

3

J

j

9

5

Q

M

z

/

E

Y

T

H

O

w

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

n

m

v

q

D

I

V

g

B

K

J

z

/

X

Q

9

S

E

>

A

C

H

c

N

8

3

W

O

o

k

U

F

j

0

w

7

G

Z

f

+

p

r

y

u

2

R

Y

d

L

5

M

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

Y

N

C

u

g

V

c

S

9

y

I

k

B

3

M

j

Z

r

/

A

>

+

D

L

F

2

p

f

7

d

v

q

R

0

G

U

m

o

Q

8

X

5

P

W

K

E

J

w

n

H

z

O

4 Methods and Results

Research question: how to sample code mixed documents?

4.1 Code Mixing Index (CMI)

We used a well-known metric to measure the extent of code switching in a document - Code Mixing Index (CMI) [Das and Gamb¨ack, 2014]. Essentially, CMI measures the presence of a dominant language in a document. Let a document d expressed with k different languages, {l1, . . . , lk}, and u neutral tokens be represented as a sequence of words: [w1, . . . ,wn]. Let L(wi) return the language of word wi (or neutral if it is a neutral token). For each language, N(lj) denotes the total number of utterances of lj in the document, i.e., N(lj) = ∑i=n i=1 I(L(wi) = lj) where I is the indicator function. The CMI of the document d, CMI(d), is measured as:

CMI(d) = ∑j=k j=1 N (lj )-maxi(N (li)) n-u . In the boundary condition, where every word in the document is a neutral token, CMI is defined as 0; hence, CMI(d) ∈ [0, 1). A low CMI value indicates minimal code switching i.e. the document is almost entirely written in the dominant language. Understandably, when k = 2, the highest possible CMI is 0.5 indicating equal presence of two component languages. When L(.) is estimated using a language identification method, we denote the estimated CMI of a document as CMI∧(d).

We now illustrate with an example: [bilkul sahi baat kahi aapne imran khan saab please please no more war only peace] (loosely translates to You've spoken the absolute truth Mr. Imran Khan, please no more war, only peace.). In this example, N(en) = 7, N(he) = 6, n = 15, and u = 2. Hence, CMI of the document is 7+6-7 15-2 = 0.46. We considered documents with CMI∧greater than or equal to 0.4 as documents exhibiting significant code mixing.

4.2 Estimating CMI

In order to sample documents with high CMI∧, we need a reliable token-level language identification module. We used the polyglot-embedding based method (denoted by ˆ Lpolyglot ) proposed by Palakodety et al. [2020a]. We chose ˆ Lpolyglot because it requires minimal supervision and is particularly well-suited for noisy social media texts [Palakodety et al., 2020c]. In particular, ˆ Lpolyglot involves obtaining the document embeddings, and then using k-means on these embeddings. The method is shown to reveal highly precise monolingual clusters. Previous use-cases of ˆ Lpolyglot were limited to document-level language identification. In our experiments we found that without any significant modification, the technique is capable of token-level language identification with considerable accuracy. Our token-level language identification follows the same method presented by Palakodety et al. [Palakodety et al., 2020a]. We consider a token as a single-word document, obtain its embedding, and assign the language of the nearest cluster center in the document embedding space.

Detecting neutral tokens. Neutral tokens are identified using a simple heuristic: for a two-language scenario, a token is marked neutral if it is approximately equidistant from the two respective cluster centers. For a given token, w, let the Euclidean distance of w from the English cluster and Hindi cluster in the comment embedding space be represented as dist(w, en) and dist(w, he), respectively. Let the distance between the two cluster centers be expressed as dist(en, he). ˆ Lpolyglot(w) = neutral iff w ∈ (Dhe ∪ Den) and |dist(w,en)-dist(w,he)| dist(en,he) ≤ ϵ.

When ϵ is set to 0.1, our method obtains the following top 20 (ranked by frequency) neutral tokens: Pakistan, he, army, media, Modi, Pak, Pakistani, Kashmir, pilot, attack, video, news, khan, jai, 2, hind, Imran, Muslim, sir, 1. These tokens broadly include proper nouns (e.g., Modi, Khan, Pakistan), numerals (e.g., 1), technical terms (e.g., video) and overloaded words (e.g., he; he translates to is in Hindi, and is the third-person singular masculine pronoun in English).

On a data set of 300 comments with gold standard tokenlevel annotation, we found that ˆ Lpolyglot performs token-level language detection with considerable accuracy. As shown in Table 2, the overall accuracy of ˆ Lpolyglot is 88.76%.

Estimating CMI. Once the reliability of token-level language identification by ˆ Lpolyglot is established, we next evaluate the reliability of the estimate for CMI (i.e. CMI∧). We first define the subset with substantial estimated code mixing, Dcm, as the following: Dcm = {d} s.t. d ∈ Dhe ∪ Den and CMI∧(d) ≥ 0.4, i.e., the document is either part of the Predicted Label English or Romanized Hindi subset and its estimated CMI is high indicating nearly equal presence of Hindi and English. As shown in Figure 2, Dcm indeed falls in the overlapping region of the English and Hindi cluster. We manually inspected and annotated 1,000 randomly sampled comments from Dcm and found that 95.9% comments exhibited code switching. We further obtained token level consensus labels for 100 randomly sampled comments each from Dcm, Den, and Dhe . Table 1 compares the ground truth CMI and CMI∧and demonstrates that we achieved a reasonable approximation of true CMI using ˆ Lpolyglot .

Table 1: CMI estimation root mean squared error.

Corpus	CMI	CMI∧	Overall RMSE
Den	0.03	0.04	0.05
Dhe	0.10	0.12
Dcm	0.38	0.45

Table 2: Confusion matrix of token-level performance evaluation of ˆ Lpolyglot on 300 annotated comments fromDhe ∪Den.

		neutral	en	he
	neutral	702	325	144
True Label	en	334	4690	56
	he	85	148	3235

4.3 Sampling Hope Speech FromDhe

Method	Performance
random-Sample(Dhe )	1.8%
NN-Sample(Dhope he )	18.59%
NN-Sample(Dhope )	26.93%
NN-Sample(Dhope he,+)	21.88%
+ )	31.68%

NN-Sample(Dhope

Table 6: Sampling performance.

Research question: How to harness the Hindi part of a code mixed document to sample hope speech from Dhe ?

Once we identify a comment subset with substantial code mixing, Dcm, obtaining hope speech comments using an offthe-shelf hope speech classifier is straight-forward. Out of 36,969 comments in Dcm, the classifier predicted a set of 199 comments, Dhope , as positives. Upon manual annotation, we obtained 149 positives (denoted as Dhope + ), i.e., 74.87% positives. Understandably, due to presence of code switching, the in-the-wild precision in Dcm is lower than previously reported in-the-wild precision of 84.68% in Den [Palakodety et al., 2020a]. Table 3 lists a subset of randomly sampled comments fromDhope + . We noted that the Hindi component of the comments were consistent with the overall sentiment of the comment.

A noisy approximation of the Hindi sub-part of these comments can be obtained by ˆ Lpolyglot through discarding nonHindi tokens.

I love India I am Pakistani mein amun chahta hon khuda ke waste jang nai peace peace peace

I love India, I am Pakistani. I want peace for God's sake, not war, peace peace peace.

For instance, the above comment is transformed into [mein amun chahta hon khuda ke jang nai] (loosely translates to I want peace for God not war) when we discard non-Hindi tokens using ˆ Lpolyglot. Waste is both a valid English and Hindi word (meaning sake), and the language detector makes an error in correctly predicting it. We admit that it is possible to use more sophisticated methods to extract Hindi that consider context (e.g., considering context to assign label to a fence word) and possibly squeeze more performance out of it. However, we are primarily interested in establishing a blue-print for harnessing code switching for social good and testing the robustness of our pipeline without resorting to performance-driven engineering. In every step of our pipeline, a better-performing algorithm (e.g., better language detection module, sophisticated method to extract Hindi, more powerful comment embeddings, further effective sampling technique) can be plugged in without disturbing the flow and with a possibility of performance improvement.

Active Sampling. Once we extract the Hindi sub-parts of Dhope (denoted as Dhope he ), our next task is to find comments in Dhe that are similar to the Hindi sub-part. To this end, we use a recently-proposed Active Sampling algorithm which samples nearest neighbors in the comment embedding space to identify rare positives [Palakodety et al., 2020c]. Our choice of this Active Sampling technique is motivated by its effectiveness in mining rare positives and reported robustness to spelling variations which is particularly critical because our corpus contains noisy social media texts and Romanized Hindi does not have standard spelling rules. Following [Palakodety et al., 2020c], we used cosine distance of the embeddings as the distance measure. Our sampling algorithm is described in Algorithm 1. This algorithm takes a seed set, S, and a sample pool U as inputs and outputs a set, E ⊂ U, containing nearest neighbors of S in the comment-embedding space. Initially, E is an empty set. At each step, we expand E with nearest neighbors that are not present in the expanded set or the seed set. The function getNearestNeighbor(c, dist) returns the comment in U with minimum distance greater than or equal to dist. The size parameter is set to 5, i.e., for each comment, we add five unique nearest neighbors. We set U to Dhe since we are interested in detecting hope speech in Hindi.

Table 3: Random sample of code mixed hope speech obtained by hope speech classifier run on Dcm.

Code switched hope speech	Loose translation
I am Pakistani agar ap dono country ny war karne ha to gurbat khatam karnay ke war karo dono countries bht gareeb hain plzz dont do war war is not solution of peace	I am Pakistani. If both countries have to wage a war, wage a war to end poverty; both countries are very poor. Please, do not war, war is not solution of peace.
please media walo nafrat phailana chhod do we want peace only jai hind	Please media folks, stop spreading hate. We want peace; hail India!
absolutely right i think ab netao kee jung ko ham dono mumalik ne rad karna hai we people of both countries want peace peace and peace	Absolutely right. I think politicians' war has to prevented by common people of both countries. We people of both countries want peace, peace, and peace.

Table 4: Random sample of hope speech obtained through NN-Sample(Dhope ).

Code switched hope speech	Loose translation
bhai ap bhi khuch rahiye allah apki har farmaish puri kare and I repeat again bhai I love you all my dear brothers and sisters mujhe ekh dusre se indian or pakistaani keh kar bulana bilkul pasand nahi hum sab bhai or bhen hai or rahenge we are good humans of earth	Brother, you also be happy. May Allah grant all your wish and I repeat again, brother, I love you all my dear brothers and sisters. I don't like to identify each other as Indian or Pakistani, we are all brothers and sisters, we are good humans of earth.
galiyan dene se kya hoga beach me to begunah awam mare gii orr ham jo jang jang krte henn kha jang itnii asan he no this war is end of the world because Ind an Pak is newclear states	Nothing will come out of abusing, we will cry for war while innocent civilians will die. Who said that war is easy? No, this war is end of the world because Ind and Pak are nuclear states.
daikhou dosto apaas me baahss bazii maat kro plz such me I have love for both Pakistan and India bus apaas me muhabaatey rakhou I m student of 9th lakn me muhabaat chataa hu donoo countries me choroo fazoul ki nafrateey love you Pakistan and my neighbour country India	Look friends, stop quarrelling among each other. Please, for real, I have love for both Pakistan and India, just harbor love between each other. I am a student of 9th grade, but I want love between both countries. Leave this useless hate, love you Pakistan and my neighbor country India.

Baselines. Recall that, a random sample of 1,000 comments from Dhe only yielded 1.8% positives which is our primary baseline method (denoted as random-Sample(Dhe )).

Table 6 compares the performance of our sampling method against the baseline (we do not explicitly mention U which is consistently set to Dhe across all NN-Sample methods). We obtained substantial improvement over the baseline. Both NN-Sample(Dhope he ) and NN-Sample(Dhope ) require human inspection only at the last step of our pipeline. Our results indicate that our approach can substantially reduce manual effort in acquiring hope speech. Effectively, we sampled hope speech from a Hindi corpus simply relying on a classifier trained on English comments and harnessing code switching as a bridge between the Hindi and English sub-corpora. In all steps of the pipeline, we perform noisy approximations in estimating CMI, extracting Hindi sub-parts of comments and of course, detecting hope speech. If we introduce little more supervision and instead expand the manually annotated hope speech set Dhope + , as expected, our performance improved. Our results indicate that using minimal manual supervision we can sample with more than 30% accuracy from the Hindi subset Dhe .

Research question: What is the benefit of extracting the Hindi sub-part? Both NN-Sample(Dhope he ) and NNSample(Dhope he,+) are outperformed by corresponding sampling methods NN-Sample(Dhope ) and NN-Sample(Dhope + ), respectively (see, Table 6). We were curious to analyze if extracting the Hindi allows sampling from the sub-region of Dhe mostly written in pure Hindi. As shown in Table 7, without remov- NN-Sample(Dhope) 0.43

Table 5: Random sample of hope speech obtained through NN-Sample(Dhope he ).

Sampled hope speech	Loose translation
jung kisi maslay ka hal nie aman qaim kro	War won't solve any problems, restore peace.
beshak India aur Pakistan ko bhaith kar baat ko suljana chahiye kyunke is ladaye me humare desh ke fouji jo bevajaah shahid ho rahe h aur Pakistan ke fouji jo bevajaah shahid ho rahe h is me na hi mantriyo ka koi nuksaan h na hi kisika ...	Of course, India and Pakistan should sit together and solve this through dialogue. In this war, ministers and others stand to lose nothing from the pointless deaths of Indian and Pakistani soldiers...
mein ek rajput hun aur hum kbhi nh chahty k donu mulk apse mein lary phely hum ek thai phir juda huwa kuch intah pasand log nhi chaty k khoon khraba na hon	I am a Rajput, I never want fight between the two countries; We were one country before the partition, only a handful of extremists want bloodshed.

Method	CMI∧
Dhe	0.12
Den	0.04
Dtrain hope	0.03
Dcm	0.44
NN-Sample(Dhe hope )	0.05
	0.43

Table 7: CMI∧comparison.

ing the non-Hindi part, NN-Sample(.) (intuitively) yielded a set with high level of code mixing. Nearest neighbors of a code switched comment are likely other code switched comments. However, sampling using just the Hindi sub-part yielded substantially less code mixing - Table 5 shows considerably less code-mixing than Table 4. Our intuition that removing non-Hindi tokens is crucial for sampling from the low CMI region of Dhe is further supported by Figure 3 wherein NN-Sample(Dhope he ) has a better spread over Dhe while NNSample(Dhope ) is mostly located in the code mixed region.

5 COVID-19 Health Guidelines Compliance

Research question: Can our method generalize to other domains? In this section, we introduce a new data set relevant to the novel COVID-19 pandemic [Johns Hopkins, 2020] and present an exploratory study on a new task - detecting comments encouraging compliance with COVID-19 health guidelines. Using minor modifications to our proposed pipeline, starting with a handful of English example comments, we show that it is possible to perform cross-lingual sampling and detect similar content in Romanized Hindi.

Data set. Our data set consists of 3,144,988 comments on 44,888 YouTube videos uploaded by 14 highly-subscribed Indian news outlets (previously used in [Palakodety et al., 2020b]) between 30 January, 2020 2 and 10 April, 2020. Using ˆ Lpolyglot, we obtained 771,035 English comments (denoted by Dcovid en ) in and 1,720,703 comments in Romanized Hindi (denoted by Dcovid he ).

Task. Our goal is to find comments in Dcovid he exhibiting compliance to health guidelines. Health guidelines were regularly revised during this period; we narrowed our focus on the following five guidelines recommended by CDC 3 (1) maintaining social distancing (2) avoiding public gatherings (3) staying home when sick (4) covering coughs and sneezes and (5) washing hands regularly. An example code-switched comment is presented below.

2 First COVID-19 positive case was reported in India on this day.

ap ka ghar se nikla ek kadam desh ke karodo logo ki kurbani pe pani dal dega so be alert and aware about our duty as citizens of India we should take oath to win against this corona and bad time

A single step out of your house will nullify the sacrifice of millions of citizens. So be alert and aware about our duty as citizens of India; we should take oath to win against this Corona and bad time.

Our work is related to Mate et al. [2020] in its shared focus of COVID-19 analysis in India; however Mate et al. [2020] investigated policy design questions, whereas we are primarily interested in mining relevant content in a low-resource language.

Unlike the previously discussed task of hope speech detection, we do not have access to a content classifier that can detect comments encouraging compliance with COVID-19 health guidelines. We instead start with a handful of example English comments specified by our annotators and aim to retrieve similar comments authored in Romanized Hindi. Our comments subset, Atarget, consists of the following five comments: [Please maintain social distancing], [Please avoid public gatherings], [Please stay at home when sick], [Please cover your coughs and sneezes], [Please wash your hands regularly]. Starting with Atarget, we aim to retrieve similar comments in Dcovid he . Note that, in this new setting, the English phrases are not sourced from the corpus but are authored by annotators. In contrast, the previous task of hope speech detection utilized a classifier to obtain the initial set of comments fromDcm.

Recall that, the pipeline for hope speech discovery in Romanized Hindi starts with a set of code mixed documents discovered by a classifier. The English tokens are discarded to formulate Romanized Hindi phrases and then the Romanized Hindi sub-corpus is queried for similar samples using these phrases. The end result is hope speech documents in Romanized Hindi. In this task, we start with annotator authored English documents encouraging compliance with guidelines. We first sample documents in the corpus semantically similar to these authored documents using NN-Sample(Atarget, Dcovid en ). This yields documents that encourage compliance authored in English and are present in the corpus. A next round of sampling is conducted to yield code mixed documents similar to these English documents (NN-Sample-En(.)). NNSample-En(.) is identical to NN-Sample(.) presented in Algorithm 1 with only one modification: it computes semantic similarity between an English document and a code mixed document by discarding the non-English tokens from the latter. At this stage, we have code mixed documents sourced from the corpus and the rest of the pipeline is similar to the previous task - any tokens not written in English are deleted, the Romanized Hindi portions are retained and used for sampling from the Romanized Hindi sub-corpus yielding comments authored in Romanized Hindi that encourage compliance with health guidelines. The added sampling phases in this particular task allow us to compensate for the lack of a classifier. We first use the annotator authored documents to obtain documents in the corpus that exhibit our desired properties and then utilize the pipeline developed earlier in this paper. Figure 4 shows the system diagram for this task.

3 https://www.cdc.gov/coronavirus/2019-ncov/prevent-gettingsick/prevention.html

Table 8: Random sample of comments obtained through our method.

Sampled comments encouraging compliance with health guidelines	Loose translation
sab log party karna band karo na kuchh din ke liye party ni karoge to ni ji paoge ka	Stop partying for a few days, will you die if you don't party?
...sirf log jagruta se hi kuch hadh tak bach sak ta hai jaise ki haath senetaiz se dhole aur musk peheny aur saaf sutra rahe...	Only public awareness can save us somewhat, for instance washing hands with a sanitizer, wearing a mask, maintaining hygiene. . .
shab e barat main ibabat apne apne gharon main hi karen ... shatan bimari ke khilaf insan ki larai ka saath den ajmer sharif	Please offer your prayers on Shab-e-baarat at home . . .Ajmer Sharif, please help in this fight against this evil disease.

For all rounds of nearest neighbor sampling, size was set to 5 yielding 625 comments sampled from Dcovid he . Upon annotation, we found 14.88% positives. A random sample of 625 comments from Dcovid he yielded 2.88% positives. Hence, even under this additional resource constraint, our pipeline obtained more than 5-fold performance improvement over a random baseline. Table 8 lists a sample of our retrieved comments that shows our pipeline obtained content authored mostly in Romanized Hindi (average CMI∧of 0.05).

6 Conclusions and FutureWork

In NLP literature, typically, code switching is viewed as an impediment to downstream analyses. In this paper 4 , we first raise a novel proposition that code switching can be harnessed for social good and human well-being. We utilise it as a bridge between a resource-rich and a low-resource language to reduce annotation efforts in the latter while leveraging resources tailored to the former. Our approach is appealing for its minimal supervision requirements. In the context of hostility diffusing hope speech comments, our methods can be used to broaden the reach of such content overcoming the varied language skills of linguistically diverse regions and transcending language barriers. In relation to the novel COVID19 pandemic, we utilize a small set of annotator authored English phrases encouraging compliance with health guidelines and retrieve similar Hindi content. Our method holds significant promise in addressing resource gaps across widely used languages. Future lines of research include (1) exploring a broader range of language pairs (2) investigating applicability in additional domains and (3) evaluating performance improvement through pipeline modifications.

Acknowledgements

We thank Rupak Sarkar for his assistance in compiling our COVID-19 data set.

This paper is dedicated to Professor Jaime G. Carbonell and his scientific contributions to the fields of Machine Learning, Natural Language Processing, and Artificial Intelligence.