MultiMirror: Neural Cross-lingual Word Alignment for Multilingual Word Sense Disambiguation

1 Introduction

In the last few years, Word Sense Disambiguation (WSD), i.e., the task of determining the meaning of a word in a given context, has witnessed a remarkable improvement in performance due to increasingly refined neural approaches [Bevilacqua et al., 2021]. However, the potential of these approaches is severely limited by the paucity of high-quality training corpora and by the demanding process, both in terms of expertise and amount of work, required to expand them. This problem, generally referred to as the knowledge acquisition bottleneck [Gale et al., 1992], has long affected several word- and sentence-level semantic fields of NLP, and is particularly relevant to WSD [Pasini, 2020], owing to the fine granularity of senses, which are often difficult to distinguish from one another, and their Zipfian distribution in a corpus.

Although several attempts have been made to address the issue, this limitation is still far from being overcome, and, particularly in languages other than English, different lines of research have been explored that drop the requirement for vast amounts of labeled data. Among these, knowledge-based approaches cope with this limitation by utilizing the linguistic information contained in largely-interconnected knowledge bases such as WordNet [Miller, 1998] and BabelNet [Navigli and Ponzetto, 2012], typically by means of graph-based strategies such as the Personalized PageRank [Agirre et al., 2014], or densest-subgraph algorithms [Moro et al., 2014]. More recently, a valid alternative to these approaches has been identified in the so-called zero-shot paradigm. Thanks to the cross-lingual representational power of recent unsupervised embeddings such as mBERT [Devlin et al., 2019], this paradigm proposes training supervised models on existing manually-annotated resources, which are typically in English, and applying them directly to other languages. Finally, a different research direction that has shown promising potential is annotation projection [Yarowsky et al., 2001]. This technique consists in propagating manually-curated labels to unlabeled data and has not only been successfully used in WSD [Scarlini et al., 2019; Barba et al., 2020], but also in several other NLP tasks, such as Semantic Role Labeling [Aminian et al., 2019] and Semantic Parsing [Blloshmi et al., 2020].

However, despite significant steps forward, current approaches still suffer from two main shortcomings: i) they often depend heavily on the availability of high-quality and wide-coverage external resources in order to validate candidate projections, thus posing a serious limitation, particularly when dealing with low-resource languages; ii) they strive to transfer annotations onto untagged text, either from a dataset of a different nature (e.g., from news to Wikipedia), or by assigning sense tags automatically with clustering or induction techniques. In contrast, we transfer sense tags across parallel sentences, arguing that this is the most natural setting for label propagation, and an affordable one for several language pairs, thanks to the recent advances in multilingual generative approaches [Tang et al., 2020].

To address the aforementioned issues, we introduce MULTIMIRROR, a novel approach for cross-lingual label projection based on an innovative neural architecture for word alignment. Our contribution is threefold:

1. We propose a cross-lingual sense projection approach based on word alignment, which enables a simple mBERT-powered classifier to achieve a new state of the art on established WSD datasets and proves to be effec-

tive also when projecting labels onto a distant language, i.e., Japanese.

1. We put forward a novel neural architecture for word alignment which is capable of aligning all source and target tokens at the same time with just a few hundred training sentences, achieving a new state of the art and considerably reducing the processing speed.
2. Additionally, as a by-product of our experiments, we also release four new manually-annotated datasets for word alignment in the following language combinations: English-French, English-German, English-Italian, and English-Spanish.

2 RelatedWork

Several efforts have been made to cope with the scarcity of sense-tagged data via annotation projection. The earliest works focused on harvesting training instances by exploiting WordNet's lexical relations and particularly monosemous relatives of polysemous words [Leacock et al., 1998; Agirre and Martinez, 2004]. Although these approaches proved their ability to produce new training instances, they failed however in scaling across languages and domains.

Subsequent research addressed these drawbacks by leveraging parallel corpora [Delli Bovi et al., 2017], or multilingual knowledge bases such as BabelNet. For instance, Pasini and Navigli [2017] proposed a language-independent approach, which, given as input a multilingual knowledge base and an unlabeled corpus, is capable of generating senseannotated data in a fully-automatic fashion. Following this line of research, Scarlini et al. [2019] automatically assigned senses to words in Wikipedia sentences building upon the One Sense per Wikipedia Category assumption, that is, all occurrences of a given word in Wikipedia pages associated with the same Wikipedia category share the same meaning. Recently, Barba et al. [2020] proposed MuLaN, a label projection approach using unsupervised multilingual embeddings to automatically generate sense-annotated data. Taking as input a labeled dataset and an unlabeled corpus, MuLaN projects all labeled and unlabeled instances into a shared vector space and propagates labels by adopting the k-nearest neighbors algorithm with an additional candidate validation step.

In other areas of NLP, cross-lingual label propagation has often been implemented via word alignment, i.e., the task of identifying translation correspondences between words in parallel sentences. For instance, Xi and Hwa [2005] automatically generated data for Chinese POS Tagging, Pad´o and Lapata [2009] exploited word alignment to project semantic roles and, more recently, Stengel-Eskin et al. [2019] proposed a neural discriminative architecture for word alignment and applied it to label propagation for Chinese NER. However, despite its success in a wide range of NLP tasks, to the best of our knowledge, no attempt has been made previously to leverage neural word alignment to project word senses across languages.

3 MultiMirror

In this Section, we describe MULTIMIRROR, our approach for cross-lingual sense projection. We first explain the novel word alignment model we put forward and compare it to the currently best-performing system (Section 3.1). Then, we detail how, using this model, we automatically generate sensetagged corpora (Section 3.2).

3.1 Cross-lingual Word Alignment Model

We now introduce our discriminative word alignment model, whose structure is reported in Figure 1. It takes as input two parallel sentences U = u1, . . . , ui, . . . , ul and V = v1, . . . , vj, . . . , vk, where ui is the i-th token of U and vj is the j-th token of V . In order to obtain continuous representations of each token, we employ a pretrained contextualized embedding, namely multilingual BERT [Devlin et al., 2019, mBERT]. Following the reference paper, we concatenate the two sentences, separating themwith a special token, i.e. [SEP], and surround the whole sequence with two additional tokens, namely [CLS] and [SEP]. In order to match mBERT input format, we further split the input tokens into subwords.

Once fed to mBERT, we take the output of its final layer and use it to compute a representation for each token by averaging the vectors associated with the subwords that token was split into. Thanks to this procedure, subwords can attend to each other and generate a representation that is not only contextualized on the sentence in which they appear, but also on its translation. Finally, we leverage an additional 6-layer Transformer Encoder, fully resembling the architecture of mBERT, 1 so as to enable a token-level contextualization; formally, it takes as input the current representations of each token and outputs a sequence of l + k vectors token level out = hu1 , . . . , hul , hv1 , . . . , hvk of dimension 768 each. 2

As we now have a fully contextualized representation for each token, we classify each possible alignment separately. To this end, we first compute the tensor H ∈ Rl×k×768 , where Hij is the vector resulting from the element-wise product of hui and hvj . Then, the word alignment matrix A ∈ Rl×k , with Aij containing the probability of aligning ui to vj, is obtained as follows:

R1 = Relu(HW1)

where W1 and W2 are both matrices ∈ R768×768 and W3 ∈ R768×1 .

We train our model by minimizing the Binary Cross Entropy loss between A and ˆ A, i.e., the alignment reference matrix containing the gold standard annotations, and such that ˆ Aij = 1 if the tokens ui and vj are aligned and 0 otherwise. The final loss for each sentence is thus computed as:

1 With the exception of the embedding layer, which we omit as we already have continuous representations for each token.

2 The hidden size of the final layer of multilingual BERT.

Finally, as the datasets we employ were annotated by different annotators and, more in general, the consistency of the annotations is hampered by human factors, we apply a special masking over the loss function. Indeed, if two tokens ui and vj in two parallel sentences are not aligned to any other token, i.e., ∄ m ∈ {1, . . . , l} | ˆ Amj = 1 and ∄ r ∈ {1, . . . , k} | ˆ Air = 1, the prediction of the model is not considered when computing the loss function; thus, the model is only updated when the predictions consider at least one token that has been manually aligned between the two parallel sentences. We train our model with a learning rate of 2 × 10-5 , Adam [Kingma and Ba, 2015] as the optimizer and a token batch size of 512.

Datasets The datasets we select for our experiments are those described by Nagata et al. [2020]. We use 4 manuallyannotated datasets for word alignment in the following language combinations: English-French (En-Fr), GermanEnglish (De-En), Japanese-English (Ja-En) and RomanianEnglish (Ro-En). The En-Fr dataset and the Ro-En dataset were proposed during the the HLT-NAACL-2003 workshop on Building and Using Parallel Texts [Mihalcea and Pedersen, 2003]. The De-En dataset is derived from [Vilar et al., 2006] and, finally, the Ja-En dataset was provided by Neubig [2011] and obtained by manually translating Wikipedia pages regarding Kyoto from Japanese into English. We split the sentences into training and test data as detailed in Nagata et al. [2020].

Comparison System We compare against the system with the best reported performances on these datasets, that is, the architecture proposed by Nagata et al. [2020]. The authors present a novel SQuAD-style approach in which every alignment is formulated as a query with respect to a context: the query represents a word in a sentence, the context is the translation of that sentence and the answer is the query word's corresponding translation. As in the authors' experimental setup, we consider here their bidi-average symmetrization strategy.

Results We show in Table 1 how our model fares when compared to our competitor in terms of F1 score over the alignments. Following previous works, we do not report the AER measure since it has been debated to be skewed towards precision [Fraser and Marcu, 2007].

As a first result, we note how our model performs considerably better in all settings and, interestingly, that the difference is most marked on a distant language, namely Japanese, where MULTIMIRROR surpasses Nagata et al. [2020] by almost 2 F1 points. Second, with the sole exception of En-Fr, our approach features a recall that is always consistently higher, with the closest gap on De-En amounting to 1.9 points. This finding is particularly relevant to our setting as MULTIMIRROR features filtering heuristics that validate the proposed alignments, thus discarding any spurious ones that might occur.

However, these results do not fully convey the benefits of our formulation. Indeed, thanks to it: i) we do not require any symmetrization strategy as our predictions are symmetrical by design; ii) we enable the alignment of non-contiguous spans, which is not straightforward in SQuAD-style formulations; iii) our architecture is significantly faster since tokens are aligned jointly and in a single pass, in contrast to Nagata et al. [2020] where, instead, a different query for each source token has to be performed.

3.2 Sense Projection

Based on the output of our word alignment model, we now show how MULTIMIRROR projects senses across languages. Taking as input a source corpus, i.e., a list of sense-tagged sentences s1 , . . . , sn in language L1 and their respective translations t1 , . . . , tn in language L2, MULTIMIRROR automatically projects sense annotations from each si to the corresponding ti , thus producing an automatically-tagged dataset as output.

Let si = si1 . . . si|si | be the i-th source sentence and ti the corresponding target sentence defined analogously. Given a sense tag la,b for a contiguous span sia . . . sib in si , 3 our aim is to identify the target span tia′ . . . tib′ onto which sense tag la,b should be projected. To this end, we utilize the alignment matrix A obtained from our cross-lingual word alignment model (Section 3.1) for the pair (si , ti ). However, the alignments in A are token-level and potentially non-contiguous. We therefore bring together such alignments to produce a list of nonoverlapping continuous target spans:

whose elements are sorted by their start index a′ . For each label la,b in the set of sense annotations for si , we select the first target span τ ∈ T(a, b) which does not overlap with any of the previously-chosen target spans for any of the preceding labels and such that the part of speech of any of the tokens in τ matches with the one of the sense la,b we are projecting.

4 Experimental Setup

In this Section, we present the experimental setting we adopt to evaluate MULTIMIRROR in the Word Sense Disambiguation task. First, we detail the corpora we use in our sense projection process to automatically create training data (Section 4.1). Second, we illustrate the test bed (Section 4.2) and the WSD reference model employed (Section 4.3). Finally, we describe the comparison systems in Section 4.4.

3 sia . . . sib corresponds to a word if a = b and a multiword expression otherwise.

4.1 Datasets

Alignment Datasets In order to perform sense projection, we need data on which to train our alignment model. To this end, we use the same reference alignment data described in Section 3.1. As those datasets cover only alignments between English and three other languages, i.e., French, German and Japanese, we produce datasets of comparable size for Italian and Spanish by manually aligning for each of them 300 sentence pairs (with around 4,000 aligned tokens) fromWikiMatrix [Schwenk et al., 2021], of which 50 are reserved for development.

However, our alignment data is based on Wikipedia text, while the reference data from Section 3.1 comes from different genres, including constitution laws, news text, etc. To determine whether different genres and domains might impact on the performance of our word alignment model (and, consequently, on WSD), we also create analogous alignment datasets in two of the languages for which reference data was already available (cf. Section 3.1), i.e., French and German.

Overall, we report slightly inferior word alignment performances when training on our newly annotated datasets and testing on the reference test sets, respectively 84.6 for French (-2.2) and 83.2 for German (-3.5). This result is partially motivated by the different source distribution of the two datasets and the smaller number of training instances caused by the fact that the sentences in WikiMatrix have a shorter length on average. In Section 5, we further investigate this difference and evaluate how both our training data and the reference training data fare when used for sense projection.

Source and Target Corpora As source corpus, following Barba et al. [2020], we use the concatenation of the two largest English corpora tagged with WordNet senses, i.e., SemCor [Miller et al., 1993] and the Princeton WordNet Gloss Corpus 4 (WNG). This concatenation amounts to roughly 720k sense annotations. As reference translations of this data are not available, we generate the target corpus by automatically translating the source corpus. To this end, we employ the multilingual translation model released by Tang et al. [2020] and use it to perform the translation towards the languages included in our evaluation test bed, namely: French, German, Italian, Japanese, and Spanish.

Sense Inventory Following Barba et al. [2020], we use BabelNet [Navigli and Ponzetto, 2012] as our sense inventory, whose multilingual synsets include WordNet ones. This choice enables a seamless projection of senses from WordNet-annotated corpora, like ours, to any other language, since the projected synset remains the same. We note that, in stark contrast to other dataset creation approaches, including MuLaN, we propagate sense annotations based solely on the resulting span alignment and independently of whether BabelNet contains the target span as a lexicalization in the synset at hand.

4 https://wordnetcode.princeton.edu/glosstag.shtml

Produced Datasets We report coverage statistics in Table 2 for the datasets generated by both MULTIMIRROR and our strongest competitor, i.e., MuLaN [Barba et al., 2020], over each of the 5 target languages we consider. As Japanese was not among the languages originally included in Barba et al. [2020], we execute their projection pipeline ourselves 5 , transferring annotations from the concatenations of SemCor and WNG towards Japanese Wikipedia 6 . First of all, we note how MULTIMIRROR steadily transfers a significantly larger number of instances, the only exception being Japanese, highlighting the better coverage our system attains. However, arguably more interesting is the difference in the number of senses and synsets transferred; indeed, not only does our approach increase the number of projected synsets by as few as 2k, but also pushes up the amount of projected senses by an astounding margin, almost quadrupling them in Japanese. This seems to suggest that, for a given synset, most transfers MuLaN performs are skewed towards a specific sense.

4.2 Test Bed

We evaluate our automatically-generated sense-tagged corpora against the well-established SemEval-13 [Navigli et al., 2013] and SemEval-15 [Moro and Navigli, 2015] datasets 7 , the former containing nominal instances in German, French, Italian and Spanish, and the latter covering all open-class parts of speech for Italian and Spanish.

Furthermore, so as to assess the extent to which our approach and its alternatives scale to distant languages, we include Japanese in our experiments and evaluate MULTIMIRROR on the Japanese section of XL-WSD, a cross-lingual Word Sense Disambiguation framework recently proposed by Pasini et al. [2021]; 8 the resources the authors released for this language include both a validation set and a test set, featuring 1,901 and 7,602 instances, respectively, and covering all parts of speech.

As in previous works, we report the F1 score between reference and predicted labels. We further show, in each setting, whether the results we attain are statistically significant with respect to the strongest competitor by performing McNemar's test [Dietterich, 1998].

4.3 ReferenceWSDModel

To carry out the evaluation, we employ a simple linear classifier on top of multilingual BERT (mBERT) [Devlin et al., 2019] as our reference model. Specifically, given as input a list of words, which we tokenize into word pieces as required by mBERT, we first encode them, taking the concatenation of the last 4 layers as the representation of each word piece, and then feed the resulting vectors into a fully-connected layer with a softmax activation function; should a text span be split

5 We utilized the original code that the authors released at https: //github.com/SapienzaNLP/mulan.

6 The authors used Wikipedia as the target corpus in all their ex-

periments.

7 We use the latest version of these datasets available at https://github.com/SapienzaNLP/mwsd-datasets.

8 The choice of Japanese is primarily motivated by the fact that it was the only language not already included in the standard evaluation suite for which we already had at disposal its alignment data.

into multiple word pieces, we use the vector associated with the first word piece as the representation of the whole span. We explore the usage of 2 different strategies at training time: i) fine-tuned (FT): where we fine-tune the whole model, with mBERT being updated along with the linear classifier, and ii) feature-based (FB): where, instead, we freeze weights and update only the linear classifier.

In all our experiments, we train our models on a single NVIDIA RTX 2080 Ti for 50 epochs, monitoring validation accuracy for early stopping (patience p = 3) and using the Adam optimizer with learning rate 2 × 10-5 . As no official validation sets exist for the 4 SemEval-13 and 2 SemEval-15 tasks, we reserve 1000 sentences from our training set and use them for development only; conversely, we use the official validation set when dealing with Japanese.

4.4 Comparison Systems

We consider as our baselines the Most Common Sense, computed as described in Pasini et al. [2021] for Japanese and Barba et al. [2020] for the other languages, and two neural settings in which our model is trained on English datasets and tasked to zero-shot over the test languages: i) ∅-shot-SemCor, where the learning procedure is performed over SemCor, and ii) ∅-shot-SemCor+WNG, where, instead, the concatenation of SemCor and WNG is used. In both cases, we use the English SemEval-2007 dataset for validation.

As competitors, we consider the following systems: UKB+SyntagNet [Maru et al., 2019], a knowledge-based approach which applies Personalized Page Rank over the WordNet graph further enriched with collocational edges, ARES [Scarlini et al., 2020], a semi-supervised approach for producing sense embeddings that lie in a space comparable to that of multilingual contextualized embeddings, and MuLaN, an annotation projection technique that currently holds the state of the art in most tasks of our experimental setting.

5 Results

We now investigate how MULTIMIRROR fares when compared with current state-of-the-art alternatives in multilingual WSD. To this end, we train our reference model in all 5 languages, in both FB and FT configurations, which we denote as MULTIMIRRORFB and MULTIMIRRORFT , respectively. Furthermore, we also report the results on the 4L setting, the new configuration introduced by Barba et al. [2020] where the concatenation of the datasets generated for the 4 European languages is used as the training set. We show the F1 scores that MULTIMIRROR attains in Table 3.

First of all, our results suggest that the word alignment data we manually annotated is indeed compliant with their reference counterpart (Section 3.1), with performances almost in the same ballpark on German and significantly better on French, most likely thanks to the increased number of correct alignments 9 ; indeed, as pointed out by Nagata et al. [2020], the reference French dataset contains a considerable number of noisy possible alignments.

More interestingly, we point out that MULTIMIRROR achieves state-of-the-art results, with MULTIMIRRORFB outperforming its main competitor, namely MuLaN, in all settings. In particular, since MULTIMIRRORFB features the very same WSD model MuLaN employs, this result clearly highlights the quality of our automatically-generated datasets. Once we switch to MULTIMIRRORFT , this trend becomes even more marked: 10 MULTIMIRRORFT outperforms all previous monolingual 11 systems in virtually all settings, with the only exception of the Spanish SemEval-15 dataset.

However, we believe that the most interesting result concerns Japanese, i.e., our distant language. This setting shows the largest gap between our approach and MuLaN, which actually performs worse than direct zero-shot itself, i.e., ∅- shot-SemCor+WNG. This finding suggests that there might be possible limitations in applying MuLaN on distant languages. Conversely, MULTIMIRRORFT scales remarkably well and outperforms ∅-shot-SemCor+WNG by more than 4 F1 points.

9 Here, we use correct alignment, as opposed to possible alignment.

10 We also explored training MuLaN data on this architecture but did not observe any significant gain.

11 Here we use monolingual as opposed to 4L.

Finally, we consider the 4L setting. Similarly to Barba et al. [2020], we report significant improvements over the other systems across the board, including outperforming ARES on the Spanish SemEval-15 dataset.

6 Conclusions

In this work, we presented MULTIMIRROR, a novel approach for cross-lingual sense projection based on word alignment. Leveraging an innovative neural aligner that requires just a few hundred sentences for its training, we tackle the creation of sense-tagged datasets in multiple languages by projecting sense annotations across the produced alignments. We find that MULTIMIRROR achieves high sense and synset coverage in many languages and that its automaticallygenerated datasets lead a simple transformer-based classifier to reach state-of-the-art results in established WSD benchmarks against both strong alternatives for silver data creation and state-of-the-art multilingual systems. Most interestingly, we report significantly better results when projecting onto a distant language, namely Japanese.

As a by-product of our experiments, we release four novel datasets for word alignment, each containing 300 sentences, in English and one of the following languages: French, German, Italian and Spanish. As future work, we plan to assess the scalability of this framework when considering comparable rather than parallel sentences.

Acknowledgments

The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 and the ELEXIS project No. 731015 under the European Union's Horizon 2020 research and innovation programme. This work was partially supported by the MIUR under the grant "Dipartimenti di eccellenza 2018-2022" of the Department of Computer Science of the Sapienza University of Rome.