Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

1 Introduction

Distributed word representations have shown impressive performances in multiple, diverse, downstream NLP applications when used as features [Mikolov et al., 2013; Pennington et al., 2014]. The learning objectives, optimisation methods as well as the lexical resources used in these word embedding learning methods vary significantly, resulting in a diverse set of word embeddings that capture different aspects of lexical semantics such as polysemy [Neelakantan et al., 2014; Reisinger and Mooney, 2010; Huang et al., 2012] and morphology [Cotterell et al., 2016]. Meta-embedding has emerged as a promising solution to combine diverse pre-trained source word embeddings for the purpose of producing meta word embeddings that preserve the complementary strengths of individual source word embeddings. The input and output word embeddings to the meta-embedding algorithm are referred respectively as the source and meta-embeddings.

The problem setting of meta-embedding learning differs from that of source word embedding learning in several important aspects. In a typical word embedding learning scenario, we would randomly initialise the word embeddings and subsequently update them such that some goodness criterion is optimised such as predicting the log co-occurrences in Global Vectors (GloVe) [Pennington et al., 2014] or likelihood in skip-gram with negative sampling (SGNS) [Mikolov et al., 2013]. Therefore, the source word embedding methods can significantly differ in their training objectives and optimisation methods being used. On the other hand, for a meta-embedding learning method to be generalisable to different source embedding learning methods, it must be agnostic to the internal mechanisms of the source embedding learning methods. Moreover, the vector spaces as well as their optimal dimensionalities will be different for different source embeddings, which makes it difficult to directly compare source embeddings.

Despite the above-mentioned challenges encountered in meta-embedding learning, it has several interesting properties. From a practitioners point-of-view, meta-embedding is attractive because it obviates the need to pick a single word embedding learning algorithm, which can be difficult because different word embedding learning algorithms perform best in different downstream NLP tasks under different settings [Levy et al., 2015]. Moreover, meta-embedding learning does not require the original linguistic resources (which might not be publicly available due to copyright issues) used to learn the source word embeddings, and operates directly on the (often publicly available) pre-trained word embeddings. Even in cases where the original linguistic resources are available, retraining source word embeddings from scratch can be time consuming and require specialised hardware.

Given a set of source embeddings, a simple yet competitive baseline for creating their meta-embedding is to concatenate the source embeddings [Bollegala et al., 2018; Yin and Sch¨utze, 2016; Goikoetxea et al., 2016]. Concatenation has been justified in prior work as a method that preserves the information contained in individual sources in their corresponding vector spaces. However, this explanation has no theoretical justification and it is unclear how to concatenate source embeddings with different accuracies and dimensionalities, or what losses are being minimised.

Contributions. For word embedding methods that can be expressed as matrix factorisations, we show that their concatenated meta-embedding minimises the Pairwise Inner Product (PIP) [Yin and Shen, 2018] loss between the sources and an ideal meta-embedding. Specifically, we show that PIP loss can be decomposed into a bias term that evaluates the amount of information lost due to meta-embedding and a series of variance terms that account for how source embedding spaces should be aligned with the ideal meta-embedding space to minimise the PIP loss due to meta-embedding. Our theoretical analysis extends the bias-variance decomposition of PIP loss for word embedding [Yin and Shen, 2018] to meta-embedding.

Motivated by the theory, we propose two unsupervised methods to learn the optimal concatenation weights by aligning the spectrums of the source embeddings against that of an (estimated) ideal meta-embedding matrix. In particular, no labelled data for downstream tasks are required for learning the optimal concatenation weights. We propose both sourceweighted and dimension-weighted concatenation methods. In particular, the dimension-weighted meta-embedding learning method consistently outperforms prior proposals in a range of downstream NLP tasks such as word similarity prediction, analogy detection, part-of-speech tagging, sentiment classification, sentence entailment and semantic textual similarity prediction for various combinations of source embeddings. The source code for reproducing the results reported in this paper is publicly available. 1

2 RelatedWork

Yin and Sch¨utze [2016] proposed 1TON, by projecting source embeddings to a common space via source-specific linear transformations. This method minimises squared ℓ2 distance between the meta and each source embedding assuming a common vocabulary. 1TON+ overcomes this limitation by learning pairwise linear transformations between two given sources for predicting the embeddings for out of vocabulary (OOV) words. Both of these methods can be seen as globally-linear transformations because all the words in a particular source are projected to the meta-embedding space using the same transformation matrix. In contrast, locally-linear meta-embedding (LLE) [Bollegala et al., 2018] computes the nearest neighbours for a particular word in each source and then represent a word as the linearly-weighted combination of its neighbours. Next, meta-embeddings are learnt by preserving the neighbourhood constraints. This method does not require all words to be represented by all sources, thereby obviating the need to predict missing source embeddings for OOVs.

Bao and Bollegala [2018] modelled meta-embedding learning as an autoencoding problem where information embedded in different sources are integrated at different levels to propose three types of meta-embeddings: Decoupled Autoencoded Meta-Embedding (DAEME) (independently encode each source and concatenate), Concatenated Autoencoded Meta-Embedding (CAEME) (independently decode metaembeddings to reconstruct each source), and Averaged Autoencoded Meta-Embedding (AAEME) (similar to DAEME but instead of concatenation use average). In comparison to methods that learn globally or locally linear transformations [Bollegala et al., 2018; Yin and Sch¨utze, 2016], autoencoders learn nonlinear transformations. Neill and Bollegala [2018] further extend this approach using squared cosine proximity loss as the reconstruction loss.

1 https://github.com/LivNLP/meta-concat

Vector concatenation has been used as a baseline for producing meta-embeddings [Yin and Sch¨utze, 2016]. Goikoetxea et al. [2016] concatenated independently learnt word embeddings from a corpus and the WordNet. Moreover, applying Principal Component Analysis on the concatenation further improved their performance on similarity tasks. Interestingly, Coates and Bollegala [2018] showed that accurate meta-embeddings can be produced by averaging source embeddings that exist in different vector spaces. Recent work [He et al., 2020; Jawanpuria et al., 2020] show that learning orthogonal transformations prior to averaging can further improve accuracy.

Meta-embedding methods have been used for creating sentence-level meta-embeddings from contextualised embeddings [Takahashi and Bollegala, 2022] Both contextualised embeddings and sentence-level meta-embeddings are beyond the scope of this paper, which focuses on context-independent (static) word-level meta-embedding and we defer the interested reader to [Bollegala and O'Neill, 2022] for a comprehensive survey on meta-embedding learning. To the best of our knowledge, we are the first to provide a theoretical analysis of concatenated meta-embedding learning.

3 Meta-Embedding by Concatenation

Let us consider a set of N source word embeddings s1, s2, . . . , sN, all covering a common vocabulary 2 V of n words (i.e. |V| = n). The embedding of a word w in sj is denoted by sj(w) ∈ Rkj , where kj is the dimensionality of sj. We represent sj by an embedding matrix Ej ∈ Rn×kj . For example, E1 could be the embedding matrix obtained by running SGNS on a corpus, whereas E2 could be that obtained from GloVe etc. Then, the problem of meta-embedding via weighted concatenation can be stated as - what are the optimal weights to concatenate E1, . . . , En row-wise such that some loss that represents the amount of information we lose due to meta-embedding is minimised?

3.1 Two Observations

We build our analysis on two main observations. First, we note that word embeddings to be unitary-invariant. Unitary invariance is empirically verified in prior work [Artetxe et al., 2016; Hamilton et al., 2016; Smith et al., 2017] and states that two source embeddings are identical if one can be transformed into the other by a unitary matrix. Unfortunately the dimensions in different source embeddings cannot be directly compared [Bollegala et al., 2017]. To overcome this problem, Yin and Shen [2018] proposed the Pairwise Inner-Product (PIP) matrix, PIP(E), of E given by (1) to compare source embeddings via their inner-products over the word embeddings for the same set of words.PIP(E) = EE⊤ (1)

If the word embeddings are normalised to unit ℓ2 length, EjEj ⊤ becomes the pairwise word similarity matrix. PIP

2 Missing embeddings can be predicted using, for example, linear projections as in 1TON+.

loss between two embedding matrices E1, E2 is defined as the Frobenius norm of the difference between their PIP matrices:

PIP loss enables us to compare word emebddings with different dimensionalities, trained using different algorithms and resources, which is an attractive property when analysing meta embeddings.

Second, we observe that many word embedding learning algorithms such as Latent Semantic Analysis (LSA) [Deerwester et al., 1990], GloVe, SGNS, etc. can be written as either an explicit or an implicit low-rank decomposition of a suitably transformed co-occurrence matrix, computed from a corpus [Li et al., 2015; Dhillon et al., 2015]. For example, LSA applies Singular Value Decomposition (SVD) on a Positive Pointwise Mutual Information (PPMI) matrix, GloVe decomposes a log co-occurrence matrix, and SGNS implicitly decomposes a Shifted PMI (SPMI) matrix.

More generally, if M is a signal matrix that encodes information about word associations (e.g. a PPMI matrix) and M = UDV⊤ be its SVD, then a k-dimensional embedding matrix E of M is given by E = U1:kDα1:k,1:k for some α ∈ [0, 1] by selecting the largest k singular values (in the diagonal matrix D) and corresponding left singular vectors (in the unitary matrix U). Setting α = 0.5 induces symmetric target and context vectors, similar to those obtained via SGNS or GloVe and has been found to be empirically more accurate [Levy and Goldberg, 2014]. The hyperparameter α was found to be controlling the robustness of the embeddings against over-fitting [Yin and Shen, 2018] (See Supplementary for a discussion of α). Given, α and k, a source word embedding learning algorithm can then be expressed as a function E = fα,k(M) = U1:kDα1:k,1:k that returns an embedding matrix E for an input signal matrix M.

Having stated the two observations we use to build our analysis, next we propose two different approaches for constructing meta-embeddings as the weighted concatenation of source embeddings.

3.2 Source-weighted Concatenation

Yin and Sch¨utze [2016] observed that it is important to emphasise accurate source embeddings during concatenation by multiplying all embeddings of a particular source by a sourcespecific weight, which they tune using a semantic similarity benchmark. Specifically, they compute the source-weighted meta-embedding, ˆesw(w) ∈ Rk1+···+kN , of a word w ∈ V using (3), where ⊕ denotes the vector concatenation.

Concatenation weights cj satisfy ∑Nj=1 cj = 1. Then, the source-weighted concatenated meta-embedding matrix, ˆEsw is given by (4).

⊕ to denotes the row-wise matrix concatenation.

3.3 Dimension-wighted Concatenation

The number of source embeddings is usually significantly smaller compared to their sum of dimensionalities (i.e. N≪ ∑Nj=1 kj). Moreover, the source-weighted concatenation can only adjust the length of each source embedding, and cannot perform any rotations. Therefore, the flexibility of the sourceweighting to produce accurate meta-embeddings is limited. To overcome these limitations, we propose a dimension-weighted concatenation method given by (5).

Here, Cj is a diagonal matrix with cj,1, . . . , cj,kj in the main diagonal. We require that for all j, ∑kj i=1 cj,i = 1. The dimension-weighted concatenated meta-embedding matrix ˆEdw can be written as follows:

Here, we have ∑Nj=1 kj(≫N) number of parameters at our disposal to scale each dimension of the sources, which is more flexible than the source-weighted concatenation. Indeed, source-weighting can be seen as a special case of dimensionweighting when cj = cj,1 = . . . cj,kj for all j.

3.4 Bias-Variance inMeta-Embedding

Armed with the two key observations in §3.1, we are now in a position to show how meta-embeddings under sourceand dimension weighted concatenation induce a bias-variance tradeoff in the PIP loss. Given that source-weighting is a special case of dimension-weighting, we limit our discussion to the latter. Moreover, for simplicity of the description we consider two sources here but it can be extended any number of sources. 3

Let us consider the dimension-weighted concatenated metaembedding ˆE = [E1C1; E2C2] of two source embedding matrices E1 ∈ Rn×k1 and E2 ∈ Rn×k2 with concatenation coefficient matrices C1 = diag(c1,1, . . . , c1,k1 ) and C2 = diag(c2,1, . . . , c2,k2 ). E1 and E2 are obtained by applying SVD on respective signal matrices M1 and M2 and are given by E1 = U(1) ·,1:k1 D(1)α 1:k1,1:k1 and E2 = U(2) ·,1:k2 D(2)α 1:k2,1:k2 . The diagonal matrices D(1) 1:k1,1:k1 = diag(µ1, . . . , µk1 ) and D(2) 1:k2,1:k2 = diag(ν1, . . . , νk2 ) contain the top k1 and k2 singular values of respectively of M1 and M2.

Let us assume the exsistence of an oracle that provides us with an ideal meta-embedding matrix E ∈ Rn×d , where d ≥ (k1 + k2) is the dimensionality of this ideal metaembedding space. E is ideal in the sense that it has the minimal PIP loss ∣∣∣∣∣∣PIP(E) - PIP(ˆE)∣∣∣∣∣∣F between ˆE created from source embedding matrices E1 and E2. Following the low-rank matrix decomposition approach, E = U·,1:dDα1:d,1:d can be computed using an ideal signal matrix M, where D1:d,1:d = diag(λ1, . . . , λd). However, in practice, we are not blessed with such oracles and will have to estimate M via sampling as described later in §3.5. Interestingly, in the case of ˆE constructed by dimension-weighted concatenation, we can derive an upper bound on the PIP loss using the spectral decompositions of M, M1 and M2 as stated in Theorem 1.

3 Indeed we use three sources later in experiments.

Theorem 1. For two source embedding matrices E1 and E2, the PIP loss between their dimension-weighted metaembedding ˆE = [E1C1; E2C2] and an ideal meta-embedding E is given by (7).

directional variance in s2

Proof. See the Appendix in [Bollegala, 2022] for the proof.

For symmetric signal matrices such as the ones computed via word co-occurrences, embedding matrices are given by the eigendecomposition of the signal matrices and with real eigenvalues. M,M1,M2 (hence their spectrums) as well as respective optimal dimensionalities d, k1, k2 are unknown but can be estimated as described in §3.5. Although we used the same α for the oracle and all sources for simplicity, Theorem 1 still holds when all α are different.

The perturbation bounds on the expected PIP losses are known to be tight [Vu, 2011]. The first term in the R.H.S. in (7) can be identified as a bias due to meta-embedding because as we use more dimensions in the sources (i.e. k1 + k2) for constructing a meta-embedding this term would become smaller, analogous to increasing the complexity of a prediction model. On the other hand, third and fourth terms in the R.H.S. are square roots of the sum of squared differences between the spectra of the ideal meta-embedding and each source embedding, weighted by the concatenation coefficients. These terms capture the variance in the magnitudes of the spectra. Likewise, the second and fifth terms compare the left singular vectors (directions) between each source and the ideal meta-embedding, representing the variance in the directions.

The bias-variance decomposition (7) provides a principled approach for constructing concatenated meta-embeddings. To minimise the PIP loss, we must match the spectra of the sources against the ideal meta-embedding by finding suitable concatenation coefficients such that the magnitude variance terms are minimised. Recall that in meta-embedding, the sources are pretrained and given, thus their spectra are fixed. Therefore, PIP loss (7) is a function only of concatenation coefficients for all c1,i, c2,j and their optimal values can be obtained by differentiating w.r.t. those variables. For example, for the first source, under dimension- and source-weighted concatenations the optimal weights are given respectively by (8) and (9).

Interestingly, directional variances can be further minimised by, for example, rotating the sources to have orthonormal directions with the ideal meta-embedding. Indeed, prior work [He et al., 2020; Jawanpuria et al., 2020] have experimentally shown that by applying orthogonal transformations to the source embeddings we can further improve the performance in averaged meta-embeddings. Alternatively, if we have a choice on what sources to use, we can use directional variance terms to select a subset of source embeddings for creating meta-embeddings.

3.5 Ideal Embedding Estimation

In meta-embedding learning, we assume that we are given a set of trained source embeddings with specific dimensionalities. Therefore, k1, . . . , kN, α that determine the source embeddings, E1, . . . , EN , obtained by applying SVD on the respective signal matrices, M1, . . . ,MN, are not parameters in the meta-embedding learning task, but are related to the individual source embedding learning tasks. On the other hand, the singular values, λ1, . . . , λd, of the ideal meta-embedding signal matrix, M, its dimensionality, d, are unknown parameters related to the meta-embedding learning task and must be estimated. In the case of concatenated meta-embeddings discussed in this paper, we have d = ∑j kj. However, we must still estimate λ1, . . . , λd as they are required in the computations of source- and dimension-weighted concatenation coefficients, given respectively by (8) and (9).

For this purpose, we model a signal matrix, ˜M, that we compute from a corpus using some co-occurrence measure as the addition of zero mean Gaussian noise matrix Z to an ideal signal matrixM, (i.e. ˜M=M+Z). Let the spectrum of ˜Mbe ˜λ1, . . . , ˜λd. Next, we randomly split the training corpus into two equally large subsets, and compute two signal matrices: ˜M1 = M+ Z1 and ˜M2 = M+ Z2, where Z1, Z2 are two independent copies of noise with variance 2σ2 . Observing that ˜M1 - ˜M2 = Z1 - Z2 is a random matrix with zero mean and 4σ2 variance, we can estimate the noise for symmetric signal matrices by σ = 1 2n ∣∣∣∣ ˜M1 - ˜M2 ∣∣∣∣F .

Then, we can estimate the spectrum of the ideal signal matrix λ1, . . . , λd from the spectrum of the estimated signal matrix using universal singular value thresholding [Chatterjee, 2012] as λi = max(˜λi - 2σ√n, 0). The rank of M is determined by the smallest (i + 1) for which ˜λi ≤ 2σ√n. Although we only require the spectra of the ideal signal matrices for computing the meta-embedding coefficients, if we wish we could reconstruct M = UDV⊤, where D = diag(λ1, . . . , λd), U and V are random orthonormal matrices, obtained via SVD on a random matrix of suitable shape.

4 Experiments

4.1 Effect of Weighted Concatenation

To evaluate the different weighting methods, we create the meta-embeddings of the following three source embeddings: GloVe embeddings trained on the Wikipedia Text8 corpus (17M tokens) 4 , SGNS embeddings trained on the WMT21 English News Crawl (206M tokens), 5 and LSA embeddings trained on an Amazon product review corpus (492M tokens) [Ni et al., 2019].

For GloVe, the elements of the signal matrix Mij are computed as log(Xij) where Xij is the frequency of cooccurrence between word wi and wj. The elements of the signal matrix for SGNS are set to Mij = PMI(wi, wj )-log β, where PMI(wi,wj) is the PMI between wi and wj. For LSA, the elements of its signal matrix are set to Mij = max(PMI(wi,wj), 0). Source embeddings are created by applying SVD on the signal matrices.

The optimal dimensionality of a source embedding is determined by selecting the dimensionality that minimises the PIP loss between the computed and ideal source embedding matrices for varying dimensionalities upto their estimated rank. The estimated optimal dimensionality for GloVe (k1 = 1707) is significantly larger than that for SGNS (k2 = 303) and LSA (k3 = 220). This is because the estimated noise for GloVe from the Text8 corpus is significantly smaller (σ = 0.0891) compared to that for SGNS (σ = 0.3292) and LSA (σ = 0.3604), enabling us to fit more dimensions for GloVe for the same cost in bias. We used the MTurk-771 as a development dataset to estimate the co-occurrence window (set to 5 tokens) and β (set to 3) in our experiments. Moreover, α is set to 0.5 for all source embeddings, which reported the best performance on MTurk-771.

We compare the proposed source-weighted (SW) and dimension-weighted (DW) concatenation methods against several baselines: unweighted concatenation (UW) where we concatenate the source embeddings without applying any weighting, averaging (AVG) the source embeddings as proposed by Coates and Bollegala [2018] after padding zeros to sources with lesser dimensionalities, and applying SVD on UW to reduce the dimensionality to 200, which reported the best performance on the development data.

4 http://mattmahoney.net/dc/textdata.html

5 http://statmt.org/wmt21/translation-task.html

Table 1 shows the Spearman correlation coefficient on two true word similarity datasets - SimLex and SimVerb, which are known to be reliable true similarity datasets [Faruqui et al., 2016]. To evaluate meta-embeddings for Part-of-Speech (PoS) tagging as a downstream task, we initialise an LSTM with pretrained source/meta embeddings and train a PoS tagger using the CoNLL-2003 train dataset. PoS tagging accuracy on the CoNLL-2003 test dataset is reported for different embeddings in Table 1. Among the three sources, we see significant variations in performance over the tasks, where LSA performs best on SimLex and SimVerb, while SGNS on PoS. This is due to the size and quality of corpora used to train the source embeddings, and simulate a challenging meta-embedding learning problem. UW remains a strong baseline, outperforming AVG and SVD, and even SW in SimLex. However, SW outperforms UW in SimVerb and PoS tasks. Overall, DW reports best performance on all tasks demonstrating its superior ability to learn accurate meta-embeddings even in the presence of weak source embeddings.

4.2 Comparisons against PriorWork

We compare against previously proposed meta-embedding learning methods such as LLE, 1TON, DAEME, CAEME, AAEME described in §2. Unfortunately, prior work have used different sources and corpora which makes direct comparison of published results impossible. To conduct a fair evaluation, we train GloVe (k = 736, σ = 0.1472), SGNS (k = 121, σ = 0.3566) and LSA (k = 119, σ = 0.3521) on the Text8 corpus as the source embeddings. For prior methods, we use the publicly available implementations by the original authors. The average run times of SW and DW are ca. 30 min (wall clock time) measured on a EC2 p3.2xlarge instance. Hyperparameters for those methods were tuned on MTurk-771 as reported in the Supplementary.

In Table 2, we use evaluation tasks and benchmark datasets used in prior work: (1) word similarity prediction (Word Similarity-353 (WS), Rubenstein-Goodenough (RG), rare words (RW), Multimodal Distributional Semantics (MEN), (2) analogy detection (Google analogies (GL), Microsoft Research syntactic analogy dataset (MSR), (3) sentiment classification (movie reviews (MR), customer reviews (CR), opinion polarity (MPQA)), (4) semantic textual similarity benchmark (STS), (5) textual entailment (Ent) and (6) relatedness (SICK). For computational convenience, we limit the vocabulary to the most frequent 20k words. Note that this is significantly smaller compared to vocabulary sizes and training corpora used in publicly available word embeddings (e.g. GloVe embeddings are trained on 42B tokens from Common Crawl corpus covering 2.8Mword vocabulary). Therefore, the absolute performance scores we report here cannot be compared against SoTA on these benchmark. The goal in our evaluation is to compare the relative performances among the different meta-embedding methods.

From Table 2 we see that the proposed SW and DW methods report the best performance in all benchmarks. In particular, the performance of DW in RW, CR and SW in GL, MSR are statistically significant over the second best methods for those datasets. Among the three sources, we see that SGNS and LSA perform similarly in most tasks. Given that both SGNS

and LSA use variants of PMI 6 this behaviour is to be expected. However, we note that the absolute performance of source embeddings usually improves when trained on larger corpora and thus what is important in meta-embedding evaluation is not the absolute performance of the sources, but how much relative improvements one can obtain by applying a meta-embedding algorithm on top of the pre-trained source embeddings. In this regard, we see that in particular DW significantly improves over the best input source embedding in all tasks, except in RG where the improvement over LSA is statistically insignificant.

We see that UW is a strong baseline, outperforming even more complex prior proposals. By learning unsupervised concatenation weights, we can further improve UW as seen by the performance of SW and DW. In particular, DW outperforms SW in all datasets, except in RG, GL and MSR. RG is a smaller (65 word pairs) dataset and the difference in performance between SW and DW there is not significant. Considering that GL and MSR are datasets for word analogy prediction, we see that SW is particularly better for analogy tasks.

SVD and AVG perform even worse than some of the sources due misaligned dimensionalities. Therefore, when the sources are of different dimensionalities the projection matrices have different numbers of parameters, requiring careful balancing. We recommend that future work evaluate the robustness in performance considering the optimal dimensionalities of the sources. To ablate sources, we consider all pairwise metaembeddings in Table 3, which shows the performance on word similarity (MEN) and multiple STS benchmarks (STS 13, 14, 15 and 16). We see that DW consistently outperforms all other methods, showing the effectiveness of dimension-weighing when meta-embedding sources of different dimensionalities and performances.

5 Conclusion

We showed that concatenated meta-embeddings minimise PIP loss and proposed two weighted-concatenation methods. Experiments showed that the dimension-weighted metaembeddings outperform prior work on multiple tasks. We plan to extend this result to contextualised embeddings.

6 when β = 1 SPMI becomes PMI