A Deep Generative Model for Code Switched Text

1 Introduction

Multilingual text is very common on social media platforms like Twitter and Facebook. A prominent expression of multilingualism in informal text and speech is code-switching: alternating between two languages, often with one rendered in the other's character set. Many NLP tasks benefit from accurate statistical language models. Therefore, extending monolingual language models to code-switched text is important.

Many state-of-the-art monolingual models are based on recurrent neural networks (RNNs) [Chandu et al., 2018; Winata et al., 2018]. We call them RNN language models or RNNlms. RNNlm decoders, conditioned on task-specific features, are heavily used in machine translation [Sutskever et al., 2014; Bahdanau et al., 2014], image captioning [Vinyals et al., 2015] textual entailment [Bowman et al., 2015a] and speech recognition [Chorowski et al., 2015].

∗Both the authors contributed equally

Training RNNlms is data-intensive. The paucity of language-tagged code-switched text has been a major impediment to training RNNlms well. This strongly motivates the automatic generation of plausible synthetic code-switched text to train state-of-the-art neural language models.

Synthetic but realistic monolingual text generation is itself a challenging problem, on which recent deep generative techniques have made considerable progress. Two generative architectures are predominantly used: (a) Generative Adversarial Networks (GAN) [Goodfellow et al., 2014] and (b) Variational AutoEncoders (VAE) [Kingma and Welling, 2013]. Several recent works have successfully extended GANs [Zhang et al., 2017; Kannan and Vinyals, 2017] and VAEs [Bowman et al., 2015b] to generate diverse and plausible synthetic monolingual texts.

Generating plausible code-switched text is an even more delicate task than generating monolingual text. Linguistic studies show that bilingual speakers switch languages by following various complex constraints [Myers-Scotton, 1997; Muysken et al., 2000] which may even include the intensity of sentiment expressed in various segments of text [Rudra et al., 2016]. [Pratapa et al., 2018] synthesized code-mixed sentences by leveraging linguistic constraints arising from Equivalence Constraint Theory. While this works well for language pairs with good structural correspondence (like English-Spanish), we observe performance degrades with weaker correspondence (like English-Hindi). [Bidisha et al., 2019] proposes a method to generate code-switched text given a source and target sentence pair, with a restrictive set of switching patterns. Therefore, a code-switched text synthesizer needs to learn overall syntax distributions of codeswitched sentences, as well as model complex switching patterns conditioned on it.

Owing to its great syntactic and switching diversity, large volumes of language-labeled code-switched text is needed to train monolingual deep generative models, which are not available. The only alternative is to train monolingual models with parallel corpora of the two constituent languages which may be relatively easily obtainable. However, training a GAN with aligned parallel corpora may not help, because it is designed to generate a sentence from a noise distribution instead of any learned latent embedding space. Using a VAE RNNlm [Bowman et al., 2015b] is more promising. Aligned parallel corpora are expected to yield similar representations for a source-target sentence pair. Therefore, a VAE decoder conditioned on this embedding may generate some code-switch text without applying explicit external force. However, it is unlikely to learn subtle connections between context and switching decisions as well as a customized VAE solution, which is our goal.

Here we present VACS, a new deep generator of codeswitched text, based on a hierarchical VAE augmented with language- and syntax-informed switching components.

• Observed language-labeled code-switched text is encoded to a two-layer compressed representation. The lower layer encodes sequential word context. Conditioned on this lower layer, the encoder models the switching behavior in the higher layer. This contrasts with existing dual-RNNlm architectures [Garg et al., 2018] that do not have any explicit gadget to model the switching behavior. Our encoder learns the two-layer representation via variational inference so that the resulting encoded representations enable our decoder to readily generate new code-mixed sentences.
• Our decoder is designed to sample a context sequence, given a switching pattern. Unlike previous RNNlms [Chandu et al., 2018] which consider context and tag generation as independent processes, the decoder of VACS first decodes a switching pattern from the switching embedding and then uses this switching pattern memory as well as the lower-layer compressed encoding, to generate a sequence of words in a restrictive fashion. context sequence. The
• During the decoding process, VACS (trivially) generates the language labels for each word in the sentence. Thus, VACS lets us synthesize unlimited amounts of labeled code-switched text, starting with modest-sized samples.

Owing to the asymmetry between word and label sequences, our encoder and decoder layers show some asymmetries tailored to code-switching, which distinguishes VACS from a regular RNN-based VAE.

Through extensive experiments reported here, we establish that augmenting scarce natural labeled code-switched text with plentiful synthetic code-switched text generated by VACS significantly improves the perplexity of state-of-the-art language models. The perplexity of the models on held-out natural Hindi-English text drops by 33.32% compared to using only natural training data. Manual inspection also shows that VACS generates sentences with diverse mixing patterns. 1

2 Background on VAEs

VAEs [Kingma and Welling, 2013] are among the most popular deep generative models. They define a decoding probability distribution pθ(x|z) to generate observation x, given latent variables z, which are sampled from a simple prior distribution pπ(z). The objective of the VAE is to learn an approximate probabilistic inference model qφ(z|x) that encodes latent factors or features z of the variation in the observed data x.

1 https://github.com/bidishasamantakgp/VACS

Distributions p and q are often parameterized using deep neural networks. We use the maximum likelihood principle to train a VAE, i.e., maximize the expected lower bound of the likelihood on observed data x ∼ D:

To represent more complex features in the latent space, multiple VAEs are stacked hierarchically [Rezende et al., 2014; Sønderby et al., 2016]. The stack of latent variables Z are designed to learn a "feature hierarchy". For a hierarchical VAE with Λ layers, the prior, encoding and decoding probability distributions are modeled as below:

The performance of the above scheme is sensitive to the design of the layers. Layers λ ≫ 1 may fail to capture extra information. Excessively deep hierarchies with large Λ may lead to training difficulties [Sønderby et al., 2016].

3 VACS: AVAE for Code-switched Text

This section gives a high-level overview of VACS, followed by details of the building blocks, highlighting key advances beyond prior art. Our focus will be on components that implement a context-based switching distribution. Later, we describe the training process and other implementation details.

3.1 Overview

We aim to design a VAE for code-switched text, which, once trained on a collection of tagged code-mixed text should be able to generate new code-mixed text from the same vocabulary. We represent a code-switched sentence S as {(wi, yi) : i = 1, . . . , |S|}, where (wi, yi) is a pair comprised of a word wi at position i and the corresponding language label yi to which it belongs. Here we consider the simple case of switching between two languages, a source language s and a target language t. Let SOS, EOS denote start and end of sentence markers. Generation of output stops when label EOS is generated. We let yi take values from {s, t, EOS}. When discrete values like wi, yi are input to networks, they are one-hot encoded. VACS is characterized by these components:

Prior: pπ(Z)

Inference model (encoder): qφ(Z|W, Y )

Generative model (decoder): pθ(W, Y |Z)

In our formulation, Z will consist of two latent encoded representations zl and zc. Here zl is the representation of language-switching behavior, which is generated conditioned on the context representation zc, which captures syntactic and structural properties of a sentence. W is the observed sequence of words and Y is the corresponding label sequence. Given our objective, a hierarchical VAE architecture is adapted for the basic formulation with suitable departures

(b) Decoder (generation)

Figure 1: The encoder and decoder in VACS. (a) Graphical model and the recurrent architecture of the encoder. (b) Graphical model and recurrent architecture of the decoder.

whenever required. The next subsections will cover in details the inference model, generative model and prior.

3.2 Encoder

Given observed labeled sentence (W, Y ), our inference model qφ defines a hierarchical probabilistic encoding Z = (zc, zl) by first learning the content, structural embedding zc of the entire sentence. Using this embedding zc along with sequentially learned language label information the inference model qφ encodes the latent switching pattern embedding zl. Figure 1 (a) illustrates the encoder. We use two distinct RNN (LSTM) cells in the encoder, rq,c and rq,l. Their corresponding recurrent states are denoted h(q,c) and h(q,l) . Input token positions are indexed as i = 0, 1, . . . , I. The recurrence to estimate zc goes like this.

Next we estimate the encoding zl.

Here, µq,c, σq,c are the mean and standard deviation for the context encoding and µq,l, σq,l are the mean and standard deviation for the switching behavior encoding distribution. N denotes normal distribution. diag(·) represents a diagonal covariance matrix. fq,c, fq,l are modeled as feed forward stages, rq,c, rq,l are designed as recurrent units. We use the subscript q (or p) to highlight if it belongs to encoder (or decoder).

Summarizing the distinction from traditional hierarchical VAE, VACS's inference module accepts inputs in both encoding layers: word sequence at the ground layer and language label sequence at the upper layer. Learning a sequence model over language labels becomes difficult (even with hierarchical encoding) if we provide both the inputs only in the lowest level, possibly by concatenating suitable embeddings [Winata et al., 2018].

3.3 Decoder

Starting from Z = (zl, zc), our probabilistic decoder generates synthetic code-switched text with per-token language ID labels, using a two-level hierarchy of latent encoding. However, unlike the conventional hierarchical variational decoder, VACS decodes a switching pattern given zl at the upper level, then conditioned on zl and the decoded tag history it generates a content distribution zc. Here we need to design a specific decoupling mechanism of zc from zl, which is not just the reverse of encoding technique. As zl has the switching information as well as the context information, we use both zl and h(y) which is the history of label decoding to decode the distribution of zc. We design the loss in such a way that tries to minimize the difference between encoding and decoding distribution of zc.

We use two distinct RNN (LSTM) cells in the decoder, rp,l and rp,c. Their corresponding recurrent states are denoted h(p,l) and h(p,c) . Output token positions are indexed o = 1, . . . , O. The feedforward network to convert h(p,l) o to a multinomial distribution over yo is called fp,l.

Decoding continues until some yO = EOS is sampled at some O, and then stops. Effectively this amounts to sampling from pθ(Y |zl). Once all labels Y = y1, . . . , yO are generated, we decode zc and start generating words w1, . . . ,wO. fp,c denotes the feedforward network to decode zc as follows.

The feedforward network fp,w converts h(p,c) o to a multinomial distribution over words from the languages indicated by y1, . . . , yO.

We initialize h(p,c) 0 = zc and w0 = SOS (15)

If yo = s, fp,c returns a multinomial distribution over the source vocabulary, and if yo = t, fp,c returns a multinomial distribution over the target vocabulary. Effectively, we have sampled W from the distribution pθ(W|Y , zc). Overall, decoding amounts to sampling from pθ(Y ,W|Z) = pθ(Y |zl) pθ(W|Y , zc).

Figure 1 (b) illustrates the decoder architecture. VACS departs from existing dual RNN architectures [Garg et al., 2018] (two RNNs dedicated to s and t) in the following two ways:

• Instead of using a softmax output of two decoding RNNs corresponding to two language generators, VACS learns to decode language labels explicitly from a latent space.
• By using tightly-coupled decoding RNNs, parameter learning in VACS becomes more effective.

This way the decoder can generate word sequence in a more controlled fashion. The recursive word decoding unit generates a word given the predicted label from the language ID decoding layer.

3.4 Prior

The latent variable zl can be sampled from the standard normal distribution:

and then reuse pθ(zc|zl) to define

3.5 Training

Given a collection of M code-switched text S(m) = (W (m) , Y (m) ) : m= 1, . . . ,M, we train our model by maximizing the evidence lower bound (ELBO), as described in Section 2. In our case, after taking into consideration the dependence between zc and zl, the ELBO can be simplified to:

Because human-labeled code-mixed text is scarce, we first train VACS with the parallel corpora specified in Section 4.1, with aligned word embeddings. Then we further tune model parameters using real code-switched data, also specified in Section 4.1. We used Adam optimiser and KL cost annealing technique [Bowman et al., 2015b] to train VACS.

4 Experimental Setup

Here, we describe the labeled data sets, baseline paradigms, and evaluation criteria, followed by the description of language models used to evaluate the utility of the synthesized

4.1 Datasets to Train Generative Models

To train the generative models, we use a subset of the (real) Hindi-English tweets collected by [Patro et al., 2017] and automatically language-tagged by [Rijhwani et al., 2017] with reasonable accuracy. From this set we sample 6K tweets where code-switching is present, which we collect into folds rCS-train, rCS-valid. 5K tweets are found labeled with only one language. These monolingual instances are converted into parallel corpora by translating Hindi sentences to English and vise versa using Google Translation API 2 , generating 10K instances. The word embeddings of the two languages are aligned. We used different charset for Hindi (Devnagari) and English (Roman) .

4.2 Baseline Generative Models

Deep generative models. To understand the difficulties of extending existing monolingual text generators to codeswitched text, we design four baselines from two state-ofthe-art generative models. [Bowman et al., 2015b] showed impressive results at generating monolingual sentences from a continuous latent space. They extended RNNlms with a variational inference mechanism. However, their model does not allow inclusion of hand crafted features like language ID, POS tag etc. Meanwhile, [Zhang et al., 2017] proposed a GAN model to generate a diverse set of sentences. Based on these, our baseline approaches are:

• pVAE: We train the network of [Bowman et al., 2015b] with the parallel corpora. The probability of generating a word is designed as a softmax over the union vocabulary. As both of the corpora are mapped to the same latent space due to aligned embeddings, we expect the model to switch language whenever it finds a word from the other language more probable than a word in the same language as the current word.
• rVAE: To further make the model learn specific switching behaviors we train the model with the real code mixed text along with the parallel data.
• pGAN: Similar to pVAE, we train the network proposed by [Zhang et al., 2017] with the parallel text corpora.
• rGAN: GAN trained with the real code-switched data along with the parallel corpora.

RNNlm based generative models. Though language models are built primarily to estimate the likelihood of a given sentence, they can also be used as a generative tool. Recently, RNN based language models have been used to generate code-switched text as well, giving significant perplexity reduction compared to generic language models that do not consider features specific to code-switching. We compare VACS against the following code-switched LMs:

• aLM: We train the system proposed by [Winata et al., 2018] using the real code-switched text and then use a word decoder and language decoder to generate synthetic texts.

2 https://translation.googleapis.com

bLM: After training the system proposed by [Chandu et al., 2018] with the real code-mixed text, we use their LSTM decoder to generate synthetic text.

4.3 Direct/Intrinsic Evaluation

Here we analyse the features like length distribution and diversity of code-switching of generated synthetic texts. We also measured one sentence level metric Code-Mixing Index (CMI) coined by [Gamb¨ack and Das, 2016], and three corpus level metrics Multilingual index (M-Index), Burstiness and Span Entropy that were introduced in [Guzm´an et al., 2017] to demonstrate how different the generated texts are from the training corpus in terms of switching.

4.4 Indirect/Extrinsic Evaluation

We will use prior methods and VACS to generate large volumes of code-switched text. These will be used to train a payload language model (as distinct from the generative model of VACS and baselines) - specifically, the character-level LSTM proposed by [Kim et al., 2016]. Each training corpus will result in a trained payload model. The various payload models will then be used to calculate perplexity [Brown et al., 1992] scores on a held-out natural code-switched corpus. The assumption is that the payload model with the smallest perplexity was trained by the 'best' synthetic text.

Training curricula. [Baheti et al., 2017] show that language models perform better when trained with an interleaved curriculum of monolingual text from both the participating languages, then ending with code-switched (CS) text, rather than randomly mixing them. We build the curriculum from the following corpora:

Mono: 2K monolingual Hindi and 2K monolingual English tweets were sampled from the dataset. We translated Hindi to English and vice versa and make a set of 8K tweets.

X-gCS: This is the generated synthetic data. We sampled 5K generated synthetic code-switched text from various generative models. Here X denotes the generative method, which is one of {pVAE, rVAE, pGAN, rGAN, aLM, bLM, VACS}.

The specific curricula we use are:

Mono, which uses no synthetic data.

gCS | Mono, first synthetic then parallel monolingual.

Mono | gCS, first parallel monolingual, then synthetic.

Here C1|C2 denotes the sequence of corpora used to train the language model. Designing multi-task losses to guard against catastrophic forgetting is left for future work.

Validation and Testing. We sample 7K instances from the original real code-switched pool for validation and 7K for testing. These are considered as scarce evaluation resources and not used in payload training.

5 Results and Analysis

In Section 5.1 we compare intrinsic properties of synthetic texts generated from various models. In Section 5.2 we compare perplexities of payload language models prepared from text synthesized by various generators. Finally, in Section 5.3, we present anecdotes about generated text and its quality. pGAN fails to generate any appreciable rate of codeswitching. Therefore, we refrain from considering it further.

5.1 Intrinsic Properties of Synthesized Text

Based on 5000 synthetic sentences sampled from different generative methods, we report the following properties.

Length. We investigate the quality of generation methods in terms of variation in length and diversity. Other than bLM, all baselines tend to produce sentences shorter than 15 words. But Figure 2 shows that real sentences have average length ∼16 and may be as long as 25 words. rGAN generates very short sentences, at most 5 words long, and pVAE and rVAE generate sentences with an average length of ∼10. For aLM and bLM average lengths are ∼10 and ∼12 respectively. VACS has a mean of ∼17 and follows the distribution of real code-switched data most closely.

CMI, M-index, Burstiness, Span-Entropy. We report the metric values of the generated corpus and the real corpus in Table 1. VACS is closest to the real corpus in terms of M-index and Span entropy, which indicates the ratio different language tokens in the generated sentences and language span distribution is closer to the real data. Though VACS produces a larger CMI and burstiness as it can produce sentences of different lengths and various switching patterns; its CMI is still smaller than GAN, aLM, and bLM and burstiness smaller than GAN and pVAE. GAN generates the highest CMI and burstiness indicating haphazard switching patterns. On the other hand, pVAE produces lowest CMI and span entropy indicating that the generated sentences are "almost monolingual" or the language spans are equal in length. rVAE produces CMI very close to real and less diverse in terms of both switching and length distribution. Along with generating Burstiness with various switching patterns GAN also has highest burstiness

5.2 Extrinsic Perplexity

Table 2 provides a comparative study on the perplexity achieved on real validation and test CS text, after training a payload language model with different curricula spanning parallel monolingual (Mono) and generated CS (gCS) text.

Obviously, a payload language model that has seen only monolingual text when training will have large perplexity on held-out real CS text, which shows a diversity of switching behavior, in terms of both syntax structure near switches and the distribution of words used in switched segments. We expect that, in the absence of real CS text adequate to train the payload model, large volumes of synthetic text will help.

Surprisingly, this does not happen for aLM-gCS, bLMgCS, and rGAN-gCS. Adding these texts to the monolingual baseline makes payload perplexity generally worse and much worse in some cases, in particular, rGAN-gCS. VAE has better success. For the gCS|Mono curriculum, pVAE-gCS improves upon the baseline, but rVAE-gCS does worse. On further investigation we found that, pVAE generates ∼80% monolingual data, this contributes to the monolingual corpus which makes the training more coherent than mixing with code-switched data with low quality. For the Mono|gCS curriculum, both pVAE and rVAE perform worse than Mono.

In sharp contrast, VACS-gCS achieves the best (smallest) perplexity in both curricula, and much smaller than the Mono baseline. This shows that synthetic text from VACS can be used effectively to supplement parallel monolingual corpora. pVAE is the second best choice.

GAN-based synthetic text performs poorly. pGAN fails to generate any plausible code-switched text as it does not get any real code-switch samples from the parallel corpora. rGAN performance is also worse than other generative models. Though [Zhang et al., 2017] avoided mode collapsing problems common in GANs, we observed that the problem prevailed for longer sentences (>10 words) when trying to train with small amounts of code-switched text. The problem persists because CS text is much more syntactically diverse than monolingual corpus, so training a GAN with a small number of real samples produces sub-optimal results.

The performance of aLM and bLM, while better than GANs, is far from VACS. These LMs are designed explicitly for code-switched languages and require language-tagged data. Hence the generative power of such models strongly depends on the size and quality of tagged training data available.

5.3 Sample Synthetic Sentences

Table 3 shows sentences generated by VACS. All sentences in a row block are sampled from the same context embedding zc, and each row corresponds to a different zl. Note that the generated sentences seem to be able to produce a similar context. Like row (a) corresponds to cricket and (b) to assembly. It learns to produce meaningful phrases most of the cases which seem reasonable syntactically; however, semantics and pragmatics are not realistic, like in monolingual synthesis.

6 Conclusion

We proposed VACS, a novel variational autoencoder, to synthesize unlimited volumes of language-tagged code-switched text starting with modest real code-switched and abundant monolingual text. We showed that VACS generates text of various lengths and switching pattern. We also showed that synthetic code-switched text produced by VACS can help train a language model which then has low perplexity on real code-switched text. We further demonstrated that we can generate reasonable syntactically valid sentences with languagetags which can be used for various downstream applications like language labeling, POS tagging, NER etc.

Acknowledgments

B. Samanta is funded by Google India Ph.D. fellowship and the "Learning Representations from Network Data" project sponsored by Intel. We would like to thank Dr. Dipanjan Das and Dr. Dan Garrette for their valuable inputs.