Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining

1 Introduction

Recent advances in neural text-to-speech synthesis (TTS) [Li et al., 2019b; Kim et al., 2021] have yielded significant improvements in naturalness and speech quality. However, the data-intensive nature and the requirement of paired text and studio-quality audio data have limited multilingual TTS systems to resource-rich languages, which are small portions of the more than 6,000 languages in the world [Gordon Jr, 2005]. To address the limitation, current research in multilingual TTS aims not only to exploit resource-rich languages [Zen et al., 2012; Li and Zen, 2016] but also to build models for low-resource languages [Prakash et al., 2019].

Previous work has addressed low-resource TTS by using untranscribed speech data with vector-quantized variational autoencoder (VQ-VAE) [Zhang and Lin, 2020] or automatic speech recognition (ASR) models [Ni et al., 2022]. Another study [Saeki et al., 2022b] has built a massively multilingual TTS model jointly using paired TTS, paired ASR, unpaired speech, and unpaired text data. However, these approaches still rely on speech data for the target languages and face the challenge of data collection, when audio recordings for these languages are hard to obtain. In this study, we focus on the use of a text-only data for multilingual TTS as shown in Fig. 1. Previous research [Wu and Dredze, 2019; Pires et al., 2019] has shown the strong cross-lingual transferability of multilingual language models such as multilingual BERT [Devlin et al., 2019] in natural language processing (NLP) tasks. By leveraging multilingual pretraining, the model can generalize to other languages, even if it has never seen the target data in those languages. Our work applies the framework of multilingual masked language model (MLM) pretraining to TTS, with the goal of achieving zero-shot cross-lingual transfer of pronunciation and prosody. Zero-shot TTS using text data enables the development of TTS systems for languages where only textual resources are available, which potentially opens up TTS to thousands of languages [Ebrahimi and Kann, 2021; Li et al., 2022].

In this paper, we propose a multilingual TTS framework that leverages unsupervised text pretraining. Fig. 2 illustrates the proposed framework. We use a typical end-to-end TTS architecture consisting of token embedding, encoder, and decoder. Our model also has a language-aware embedding layer, which includes the token embedding layer, a language embedding layer, and a bottleneck layer. As shown in Fig. 2(a), we first pretrain the language-aware embedding layer and the encoder of the TTS model with multilingual text data. We then fine-tune the encoder and decoder of the TTS model with paired data, while the language-aware embedding layer is frozen, as illustrated in Fig. 2(b). This allows zeroshot TTS for a language not included in the paired data but present in the text data, as shown on the right in Fig. 2(c).

Our contributions are as follows. 1) We propose a zero-shot multilingual TTS framework that achieves highly intelligible TTS for an unseen language, resulting in a character error rate of less than 12%. 2) Our method also improves TTS for seen languages, resulting in byte-based models without graphemeto-phoneme (G2P) modules that outperform the phonemebased baselines. 3) Our ablation studies provide additional insights, including the effectiveness of the frozen languageaware embedding layer. The experiments were conducted on public datasets and the implementation is available 1 . We encourage readers to listen to our audio samples 2 .

2 Method

Our model has a typical neural TTS model architecture consisting of token embedding, encoder, and decoder. First, we use MLM pretraining with multilingual text data to learn cross-lingual representations. Then we perform supervised learning with paired data to learn the mapping from linguistic features to speech features. The model performs inference even for languages that are not present in the paired data.

2.1 UnsupervisedMultilingual Text Pretraining

Fig. 2(a) illustrates the unsupervised pretraining method. It uses multilingual text data consisting of languages that are not included in the paired data. Let X = (xn ∈ V |n = 1, · · · ,N) denote the input text token sequence of length N, where V denotes a vocabulary constructed for pretraining.

1 https://github.com/Takaaki-Saeki/zm-text-tts

2 https://takaaki-saeki.github.io/zm-tts-text demo

We define Dtext as the text dataset. Let Ltext denote the set of language IDs included in Dtext. First, the masked token sequence Xm and a language ID ltext ∈ Ltext are fed to the model. Let the token embedding sequence and language embedding be Z m = (zmn ∈ Rd |n = 1, · · · ,N) and el ∈ Rd , respectively. The embedding layers output Z m and el as:

where θT and θL denote the model parameters of the token embedding and language embedding layers, respectively. Then the token and language embeddings obtained in Eq. (1) are added and fed to a bottleneck layer to project them into a hidden input vector. Let Hin = (hin,n ∈ Rd |n = 1, · · · ,N) and Hout = (hout,n ∈ Rd |n = 1, · · · ,N) denote hidden vectors in the encoder input and output, respectively. Then the conditional probability p(X|X-Π) is computed as:

where θB, θE, θP denote the model parameters of the bottleneck layer, the encoder and a prediction network, respectively. In Eq. (4), Softmax(·) denotes a softmax function. We define the network with the model parameters {θB, θT, θL} as language-aware embedding layer, which jointly embeds the token sequence X and the language ID ltext as in Eq. (1) and (2). Let Π = (πk ∈ N|k = 1, · · · ,K) be the indexes of the masked tokens of length K. With the probability computed in Eq. (4), the training objective can be defined as:

We use UTF-8 bytes or International Phonetic Alphabet (IPA) symbols for the input token sequence X. For each token type, the vocabulary V is constructed fromDtext, which includes a start/end of sentence token ([SOS/EOS]). We extracted International IPA sequences using an open-source toolkit 3 . To obtain the masked token Xm, we use the same masking ratio and category as in the original BERT pretraining [Devlin et al., 2019] for each token type. Randomly, 12 % of the tokens are replaced with the [MASK] token, and 1.5 % of them are replaced with random tokens. Also, 1.5 % of the tokens are left unchanged and Lmlm is computed as in Eq. (5) for those 15 % of tokens that have indices Π.

2.2 Supervised Learning with Paired Data

Fig. 2(b) illustrates the supervised learning of the TTS model with paired data. We define the paired data and the set of language IDs as Dpaired and Lpaired, respectively. Note that we assume Lpaired ⊂ Ltext. Let Y = (yt ∈ RD |t = 1, · · · , T) denote the speech feature sequence with the length of T. We first initialize the model parameters {θE, θB, θT, θL} with those obtained in the pretraining described in § 2.1. Let θD denote the model parameter of the decoder. The speech features are predicted with teacher forcing as:

3 https://github.com/espeak-ng/espeak-ng

where Z is the unmasked token embedding sequence. Note that the unmasked token sequence is used in Eq. (6), while the masked token sequence is used in Eq. (2) Let Ltts( ˆ Y , Y ) denote the training objective of the TTS model. Then we consider two types of schemes.

Updating language-aware embedding layer. We only freeze the parameter of the language embedding layer θL while updating the rest of the parameters. Therefore the trainable model parameters can be written as

Previous work has confirmed that multilingual BERT has high cross-lingual transferability for various NLP tasks [Wu and Dredze, 2019]. This scheme corresponds to a simple finetuning of BERT [Wu and Dredze, 2019], which updates all the parameters during training for the downstream tasks 4 .

Freezing language-aware embedding layer. We freeze the bottleneck layer and the token embedding layer along with the language embedding, updating the encoder and decoder. The training process can be written as

In contrast to the scheme represented in Eq. (8), the scheme in Eq. (9) preserves the parameters of the language-aware embedding layer to facilitate cross-lingual transfer. In the evaluation, we use the scheme formulated in Eq. (9), except for the ablation study in § 3.4.

2.3 Inference

Let Lsyn denote the set of language IDs used for inference. The text token sequence X and the language ID lsyn ∈ Lsyn are fed to the model as in Eq. (1), and the encoder output is predicted as in Eq. (6). Unlike Eq. (7), the speech features are predicted as:

The output waveform is obtained by feeding the predicted features ˆ Y to a pretrained neural vocoder.

Fig. 2(c) illustrates the inference process. The left and right sides of the figure show the typical multilingual TTS and our zero-shot TTS. Previous work [Li et al., 2019a] has typically assumed seen languages, and the inference is performed with the language IDs Lseen ⊂ Lpaired. However, it is challenging to perform TTS for unseen languages Lunseen ∩ Lpaired = ∅. While other work [Saeki et al., 2022b] has built a massively multilingual TTS model that even achieves zero-shot TTS from ASR data, it uses paired data for the target languages.

4 We freeze the language embedding layer to address the mismatch between language embedding of seen and unseen languages.

Our work attempts to only use the linguistic knowledge to improve the zero-shot TTS. Thus, the inference process is written as L′unseen ∩ Lpaired = ∅ and L′unseen ⊂ Ltext. In the evaluation, we denote the inference with Lunseen and L′unseen as Fully zero-shot TTS and Text-seen zero-shot TTS, respectively. Fully zero-shot TTS performs zero-shot TTS without pretraining as in the IPA-based previous method [Staib et al., 2020], which is the baseline method in our evaluations.

2.4 Model Architecture

Our model is an autoregressive TTS model based on Transformer TTS [Li et al., 2019b], which has also been used in the previous work on byte-based multilingual TTS [He et al., 2021]. During the supervised learning described in § 2.2 and inference described in § 2, we use x-vector [Snyder et al., 2018] for the speaker embedding and add it to the encoder output through a projection layer. During supervised learning, we use the average x-vectors computed from the training data. For evaluation purposes, we perform zero-shot synthesis with the average x-vector from the test data of the target language and feed it to the model. Note that we also conduct the evaluation with x-vectors from seen languages.

For the bottleneck layer with θB, we use a residual network consisting of Layer Normalization [Ba et al., 2016], down projection, ReLU [Nair and Hinton, 2010], and up projection with the residual connection, which is used in previous work on language adaptation [Bapna et al., 2019].

3 Experimental Evaluations

3.1 Experimental Setting

Dataset

We carried out all the evaluations with publicly available datasets. Table 1 shows the sizes of the data for each language. For the unsupervised text pretraining described in § 2.1, we used transcripts from VoxPopuli [Wang et al., 2021], M-AILABS [Munich Artificial Intelligence Laboratories GmbH, 2017], and CSS10 [Park and Mulc, 2019], resulting in a total of about 2.8 GB of spoken text across 19 languages. We used CSS10 for the supervised learning described in § 2.2, and we selected seven European languages as the seen languages, with Spanish as the unseen language. The paired data consisted of one speaker per language. It should be noted that Spanish is not actually a low-resource language, but we chose to use it for evaluation purposes in order to 1) compare our zero-shot TTS methods with the oracle methods using the paired data for the target language and 2) ensure a sufficient number of evaluators for the subjective evaluation. We used 5 and 100 utterances as dev and test sets, respectively, with the remaining data used for training.

Training Details

The sampling rate was set to 16 kHz. An 80-dimension of mel filter bank, 1024 samples of FFT length, and 256 samples of frame shit were used for speech analysis. For the pretraining described in § 2.1, we trained the model for 1.2M iterations using the Noam optimizer [Vaswani et al., 2017] with the learning rate and warm-up step set to 1.0 and 10000, respectively. For the TTS model described in

§ 2.4, we used a 6-block Transformer encoder [Vaswani et al., 2017] and a 6-block Transformer decoder, with a postnet consisting of five convolutional layers with a kernel size of five. The attention dimension and the number of attention heads were set to 512 and 8, respectively. For the bottleneck layer described in § 2.4, we set the hidden dimension after the down projection to 256. The PredictionNet in Eq. (4) consisted of a linear layer, a GELU activation function [Hendrycks and Gimpel, 2016], Layer Normalization, and a linear layer with the hidden dimension of 512. We also used guided attention loss [Tachibana et al., 2018] to improve the training efficiency. For the supervised learning described in § 2.2, we trained the models for 2.47M iterations (200 epochs). The Noam optimizer was used with the warm-up step of 50000. For the neural vocoder, we trained HiFi-GAN [Kong et al., 2020] for 2M iterations with LibriTTS [Zen et al., 2019], VCTK [Veaux et al., 2017], and CSS10. For the x-vector described in § 2.4, we used a model trained on VoxCeleb1 and VoxCeleb2 [Nagrani et al., 2017] published in SpeechBrain [Ravanelli et al., 2021]. We used ESPnet2-TTS [Watanabe et al., 2018; Hayashi et al., 2021] for the implementation.

Baselines

We developed baseline models without the pretraining.

Seen language. Monolingual: We trained amodel for each language independently. Our preliminary study found that Transformer TTS was unstable 5 and could not synthesize intelligible speech in the monolingual condition due to the lack of training data. Therefore, we used Tacotron2 [Shen et al., 2018] only for the monolingual models, as in the original paper of the dataset [Park and Mulc, 2019]. Multilingual w/o LIDs: We trained a multilingual Transformer TTS model using the paired data shown in Table 1 without language IDs

5 The original paper [Li et al., 2019b] also reports the instability.

(LIDs). Multilingual w/ LIDs: We trained a multilingual Transformer TTS model with the paired data of the unseen language. It also used the language IDs.

Unseen language. We compared Fully zero-shot TTS and Text-seen zero-shot TTS defined in § 2.3. In Oracle, we used the Monolingual and Multilingual w/ LIDs, which used the paired data of the unseen language. In Fully zero-shot TTS, we used Multilingual w/o LIDs to synthesize speech from text tokens in the unseen language. This method corresponds to the conventional multilingual TTS model using bytes [He et al., 2021] or IPA symbols [Staib et al., 2020].

Evaluation Metrics

To objectively measure the synthetic speech quality, we used mel cepstral distortion (MCD) [Fukada et al., 1992] with the mel cepstrum dimension set to 25. We also evaluated the intelligibility using CERs computed with a multilingual ASR model [Radford et al., 2022]. We used a pretrained large model that is publicly available 6 . To evaluate the naturalness, we carried out listening tests to calculate five-scale mean opinion scores (MOS) of synthesized speech for each method. Forty native speakers were recruited through Amazon Mechanical Turk [Paolacci et al., 2010] for each of the tests. Furthermore, we leveraged a publicly available automatic MOS (AMOS) prediction model [Saeki et al., 2022a] to evaluate the naturalness. Note that the model was trained on English and Chinese datasets, but previous work [Seki et al., 2022] has reported that it also showed a correlation coefficient higher than 0.8 for another language (Japanese).

3.2 Evaluation Results on Seen Languages

We evaluated our framework on the seen languages included in the paired data, as defined in § 2.3. Table 2 lists the results in MCD and CER. Lower values are better for both metrics. As we can see, the byte-based or IPA-based models with the proposed multilingual pretraining performed the best across all languages and metrics. Among the baselines, byte-based monolingual and multilingual models tended to have higher MCD and CER than IPA-based models, and failed to synthesize intelligible speech in some languages. For example, the baseline byte-based models showed the high CER values for French, which has a deep orthography, meaning that a single character has different pronunciations depending on the context. We observed that our method improved the byte-based models and they outperformed the IPA-based baseline models for all the metrics and languages. It is worth noting that the proposed byte-based models even outperformed the proposed IPA-based models except for el and ru. These results suggest that our framework is effective in building a TTS model for languages without G2P modules.

3.3 Evaluation Results on Unseen Language

We evaluated our method on zero-shot TTS for the unseen language defined in § 2.3. As described in § 2.4, we first used the x-vector from the es speaker to compute the MCD. Table 3 lists the results. The baseline models showed the CERs of over 40% and MCDs of over 10.0. However, our proposed text preraining improved the metrics, resulting in CERs of less than half for both byte and IPA-based methods. Also, in contrast to the results for the seen languages, the IPA-based model outperformed the byte-based one in terms of CER. Compared with the oracle case with the paired data of the unseen language, our proposed zero-shot TTS showed higher MCD and CER but achieved only 1% difference in CER compared to the oracle byte-based monolingual model. These results demonstrate the effectiveness of our method in achieving intelligible zero-shot TTS for the unseen language.

6 https://github.com/openai/whisper

To investigate the case where the target speaker information is completely unavailable, we also used the x-vector from a seen language. We chose the fr speaker because es and fr are both categorized as Western Romance in Glottolog [Hammarstr¨om et al., 2021]. Table 3 lists the results. Note that this case does not have the MCD results, since a different speaker than the ground-truth speech was used. We can see that the unsupervised text pretraining also improved the zeroshot performance when using the x-vector from the fr speaker. In the proposed byte-based model, the cross-lingual x-vector showed the lower CER. This might result from that the es x-vector was not present in the training data whereas the fr x-vector was present in the training data.

3.4 Ablation Study

To further evaluate our method, we conducted several ablation studies. Table 4 lists the results. Bytes multilingual represents the byte-based proposed method in the evaluation of § 3.2 and 3.3. Note that it used the frozen language-aware embedding layer as formulated in Eq. (9). Some additional studies of our method are also presented in the Appendix.

In W/o bottleneck layer, we excluded the bottleneck layer and simply added the token and language embedding to obtain the encoder input in Eq. (2). We found that removing the bottleneck layer led to a performance drop in all the languages and metrics, with an average increase of 0.53 in MCD and 4.16% in CER. The largest increase was observed in the unseen language, with an increase of 1.21 in MCD. This suggests that the bottleneck layer, which projects the token and language embedding into the hidden input text representation with nonlinear dimensionality reduction, is effective in improving the generalization for zero-shot TTS.

We also evaluated the effect of including language IDs in the proposed method by comparing it with a version that excluded language IDs, referred to as W/o language ID. It corresponds to a simple multilingual BERT pretraining [Wu and Dredze, 2019] that uses only text tokens across different languages. We observed that the use of language IDs led to an Table 4: Ablation studies on training and model configurations. Bold indicates best metrics on average (Avg.).

4.5

1

average improvement of 0.5 MCD and 4.48% CER, indicating the effectiveness of our approach in using language IDs.

In W/o initializing encoder, we did not initialize the encoder θE before the supervised leaning described in § 2.2. Instead, we only initialized the parameters θT, θL, and θB with the parameters pretrained in § 2.1. Through this evaluation, we investigated whether the performance gain with our method resulted from the initialization of the languageaware embedding layer or the encoder. We observed that W/o initializing encoder resulted in an improvement of 0.04 in MCD and only a 2.27% increase in CER on average, suggesting that our method benefits more from the pretraining of the language-aware embedding layer than from the encoder.

In Updating language-aware embedding layer, we updated the language-aware embedding layer during supervised learning, as formulated in Eq. (8). We observed that freezing the language-aware embedding layer led to better performance for most languages and metrics, resulting in an average difference of 0.29 in MCD and 1.04% in CER.

3.5 Dependency on Unseen Languages

We conducted evaluations on the zero-shot TTS for different unseen languages. The eight European languages included in the paired data are composed of Indo-European and Uralic language families defined in Glottolog [Hammarstr¨om et al., 2021]. In this evaluation, we selected de and hu from each of the families. During supervised learning in § 2.2, we excluded the paired data for each of de and hu and instead included the paired data for es. Table 5 lists the results. We chose the IPA-based baseline method, which had shown better results in § 3.3. We observed that the pretraining improved the CER by around 10% and MCD by around 0.3 for de. However, the improvement in CER for hu was limited to 2%, while the MCD was improved by around 0.5. These results suggest that the performance of our zero-shot TTS is language dependent, as observed in previous work on crosslingual transfer for NLP tasks [Wu and Dredze, 2019].

Fig. 3 visualize the token embedding Z and encoder inputs Hin averaged on each utterance. We used a t-distributed

de stochastic neighbor embeddings (t-SNE) [van der Maaten and Hinton, 2008]. We observed overlaps in the token embedding for (es, fr) and (de, nl), which are classified as Western Romance and West Germanic in Glottolog, respectively. The encoder inputs are separated in the embedding space for each language. The results in Table 5 and the visualization suggest that the cross-lingual transfer works better when similar languages sharing the token embedding space are present during supervised learning. However, for languages with distinct token and language embeddings, the cross-lingual transferability might be limited. We leave the further analysis on language dependencies as a topic for future research.

3.6 Subjective Evaluations on Naturalness

We conducted evaluations on naturalness as described in § 3.1. Fig. 4 shows the results for seen languages. Note that we conducted the listening tests for de and fr. For each language, either of the proposed methods showed the highest MOS, while we did not observe any significant difference between the proposed methods and the best baseline method, which was the IPA-based multilingual model with LIDs. To further validate our results, we also evaluated the naturalness with an AMOS prediction model, as shown in Fig. 4. We observed that the either of the proposed methods showed the highest scores in all the languages. On average, the byte-based and IPA-based proposed models showed 2.89 and 2.84, respectively, while the best baseline method obtained 2.83 7 . Additionally, we observed that the byte-based proposed model often scored higher than the IPA-based proposed models, which is consistent with the results in Table 2.

Fig. 5 shows the results for unseen languages. The oracle methods had the highest MOS of 3.76 and 3.96, and the baseline zero-shot method had the lowest MOS of 3.29. The proposed methods outperformed the baseline method, and the byte- and IPA-based models had the MOS of 3.44 and 3.32, respectively. The AMOS results were consistent with the listening test results, with the proposed zero-shot TTS methods outperforming the baseline method. In this evaluation, the proposed byte-based model scored 3.21 on the AMOS, while the oracle IPA-based model scored 3.20. To further validate the results, we conducted a preference AB test on naturalness with 25 rators. As shown in Fig. 5, our byte-based model significantly outperformed the baseline IPA-based model.

4 RelatedWork

Multilingual TTS. While previous work on multilingual TTS has primarily focused on resource-rich languages [Zen et al., 2012; Li and Zen, 2016], there is growing interest in developing TTS models on low-resource languages. Several studies have explored the input tokens shared across languages such as bytes [Li et al., 2019a; He et al., 2021], IPA symbols [Gutkin, 2017], and articulatory features [Lux and Vu, 2022], to transfer knowledge from resource-rich to lowresource languages. Grapheme tokens can eliminate the perlanguage G2P knowledge, and previous work has built a bytebased TTS model for around 40 languages [He et al., 2021]. There has been work using the phonological features derived from IPA to achieve the zero-shot TTS [Staib et al., 2020]. Our framework achieves the zero-shot cross-lingual transfer with bytes by leveraging multilingual text pretraining. There have been studies on using untranscribed speech data for lowresource scenarios by leveraging VQ-VAE [Zhang and Lin, 2020] or an ASR model [Ren et al., 2019; Ni et al., 2022]. Other work [Saeki et al., 2022b] has trained a massively multilingual TTS using paired TTS, paired ASR, unpaired speech, and unpaired text data. While it also performs textonly training as in our work, it still uses the paired speech-text data of the target languages. Our framework is simple and scalable, while pioneering a novel paradigm with the zeroshot TTS approach that relies only on text data.

7 The AMOS tended to be lower than the MOS. While the MOS prediction model has a high correlation, it may produce errors in predicting absolute values, as reported in previous work [Saeki et al., 2022a]. The relative relationships are more reliable in the AMOS.

Cross-lingual representation learning for NLP. There have been studies on learning cross-lingual representations that can be applied to various NLP tasks in different languages [Gouws et al., 2015; Ruder et al., 2019]. Recent work has highlighted the strong cross-lingual transferability of multilingual BERT [Devlin et al., 2019], which has been observed to perform surprisingly well when transferred to other languages [Wu and Dredze, 2019; Conneau and Lample, 2019]. Building on this, our work leverages multilingual MLM pretraining for TTS, which improves byte-based TTS models without G2P knowledge and achieves zero-shot TTS.

Language model pretraining for TTS. Previous research has explored self-supervised text pretraining techniques for TTS. BERT models have been used to extract contextual embeddings and enhance the prosody of TTS [Hayashi et al., 2019; Xu et al., 2021]. Other studies have used phonemes jointly with graphemes [Jia et al., 2021] or subphonemes [Zhang et al., 2022] as the inputs of the MLM pretraining. Our work proposes multilingual MLM pretraining for TTS using text tokens shared across languages, rather than focusing on monolingual pretraining.

5 Conclusions

We presented a multilingual TTS framework that leverages unsupervised text pretraining. Our framework achieved highly intelligible zero-shot TTS for an unseen language, resulting in a CER of less than 12%. It also improved the TTS for seen languages, with byte-based models without G2P modules outperforming the IPA-based baselines. Our ablation studies provided additional insights, including the effectiveness of the frozen language embedding layer.

Limitations and future work. Our proposed framework has limitations. The performance gap remains between the oracle models and our zero-shot TTS models in terms of intelligibility, speech quality, and naturalness, as seen in the evaluation in § 3.3 and § 3.6. Further studies are needed to improve our zero-shot TTS. Our framework also has a limitation with language dependency, as the results in § 3.5 suggest that this dependency is caused by the presence of similar languages during supervised learning. Our future work will focus on studying this language dependency further and developing a method that performs better for various languages.

Acknowledgments

Part of this work was supported by JSPS KAKENHI Grant Number 21H05054, 22H03639, and 22J12040. This work used the Bridges system [Nystrom et al., 2015], which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center. We would like to thank the research teams at Google through the internship of the first author for providing various insights on this topic.