Unsupervised and Few-Shot Parsing from Pretrained Language Models (Extended Abstract)

1 Introduction

Automatically parsing a sentence to unveil the latent syntactic structure of the sentence is a long-standing task in natural language processing. Supervised syntactic parsing usually requires manually-annotated syntactic trees from a large treebank for training. However, building a treebank like PTB [Dahlmeier et al., 2013] is expensive and timeconsuming. Therefore, Unsupervised constituent parsing, which learns underlying structures without using any annotated trees, becomes an alternative to supervised syntactic structure learning [Shen et al., 2018a; Shen et al., 2019; Drozdov et al., 2019; Kim et al., 2019b; Kim et al., 2019a; Wang et al., 2019].

In this paper, we propose a new framework for unsupervised constituent parsing. First, we define a new syntactic distance for Unsupervised constituent Parsing, which is calculated according to an Outside Association score solely based on self-attention weight matrix: UPOA. Previous findings from many probing studies suggest that pretrained language models [Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019] are able to learn and embed latent syntactic structures in their parameters in an implicit way [Tenney et al., 2019; Jawahar et al., 2019]. Therefore, in UPOA, we exploit the self-attention weight matrix in BERT [Devlin et al., 2019] to uncover latent syntactic structures of sentences. Particularly, we estimate the syntactic distance between two adjacent words as the negative self-attention weight between two adjacent spans that consume the two words. With the estimated syntactic distance, UPOA splits a span at the split point with the largest syntactic distance. However the syntactic distance defined in UPOA only considers the association between adjacent spans (outside association), ignoring the association among words inside a span (inside association). Therefore, we further propose an enhanced Unsupervised Parsing model UPIO, which splits a span according to the strength of both Inside and Outside association. The inside association is also estimated with the self-attention weights. With the estimated inside association and outside association, we can build a syntactic tree for a sentence with a greedy or chart-based parsing algorithm.

∗The full paper is published in Artificial Intelligence 305 (2022) 103665.

Second, we extend the UPIO to FPIO, a few-shot version of UPIO that can learn substantially better syntactic structures to narrow the performance gap between unsupervised and supervised parsing with just a few annotated trees. The fundamental idea behind FPIO is based on our finding with UPIO that the two linear projection matrices used by the query and key in the self-attention mechanism have a great impact on the parsing accuracy of the FPIO. We therefore freeze other parameters in BERT and propose a method to retrain (i.e., finetune) the two projection matrices on a few annotated trees. Similarly, we extend the UPOA to FPOA, a few-shot version of UPOA.

We carry the unsupervised and few-shot parsing experiments on the Penn Treebank (PTB) [Dahlmeier et al., 2013] and SPMRL [Seddah and others, 2013] dataset. Our contributions can be summarized as follows:

• We propose an unsupervised constituent parsing model UPOA that calculates an out association score for span segmentation solely based on the self-attention weight matrix in a pretrained language model, and further propose an enhanced model, UPIO, which exploits both inside and outside association scores for estimating the likelihood of a span.
• We further extend our unsupervised parsing models UPOA and UPIO to few-shot learning methods FPOA and FPIO, which, trained on just a few annotated examples, substantially outperforms the few-shot and supervised parsing methods on the Penn Treebank and most languages of SPMRL.

• We conduct experiments and in-depth analyses on the proposed unsupervised and few-shot parsing models, which not only demonstrate the effectiveness of both models, but also provide interesting findings, e.g., those with few-shot parsing on different layers and selfattention heads.

2 Unsupervised Parsing FromPretrained Self-Attention

In this section, we introduce two unsupervised parsing methods (UPOA and UPIO) to compute split scores based on the self-attention weights from a pretrained language model.

2.1 Estimating Split Scores Based on Syntactic Distance

We first introduce the syntactic distance presented by Shen et al. [2018a] and propose a newapproach to calculating syntactic distance for unsupervised parsing from a distance matrix.

According to Shen et al. [2018a], the syntactic distance of two leaves is defined as the height of the lowest common ancestors of these two leaves. For the unlabeled constituent tree in Figure 1a, the syntactic distance of leaf "The" and leaf "government" is 2, while the syntactic distance of leaf "government" and leaf "'s" is 1. Shen et al. [2018b] propose a top-down parsing method that splits a span according to the syntactic distance between two adjacent leaves. A span is split between two adjacent words with the largest syntactic distance. The large syntactic distance indicates the two leaves share few common ancestors.

The syntactic distance defined in the aforementioned way shares some common properties with the dot product of two vectors. Firstly, syntactic distance measures the dissimilarity between two words according to the number of their common ancestors in the tree, while the dot product can be used to measure the similarity between two words with their vector representations. Secondly, a leaf node has the smallest syntactic distance with itself, while a vector has a high dot product with itself. Therefore we use the dot product of vectors to approximate the negative syntactic distance 1 of two words. The self-attention matrix of every head of a pretrained language model (particularly BERT used in this paper) contains normalized dot products of different word representations. Therefore every attention matrix can be taken as an approximation to the negative syntactic distance matrix.

1 The negative syntactic distance is not a negative number, it is the number of common ancestors of two leaves. The smaller the syntactic distance is, the more ancestors the two leaves share.

If two spans have no intersection, the syntactic distance from any word in one span to any word in the other span is the same. For example in Figure 1b, the syntactic distances from any word in the span "The government 's action" to any word in the span "was unusual" are all 4. Although we can take the negative attention weight of two adjacent words as the syntactic distance and apply the top-down algorithm proposed by Shen et al. [2018b] for unsupervised parsing, it is better to exploit the attention weights between two adjacent spans, instead of adjacent words, to estimate the syntactic distance, which can reduce the estimation bias of of syntactic distance. Given two adjacent spans: span(x, y), span(y + 1, z), we average the negative attention weights from words in one span to words in the other span as the syntactic distance between these two spans:

where aij is the attention weight that word i attends to word j. We can take the syntactic distance between span(x, y) and span(y + 1, z) as the split score of split point y in span(x, z), and parse a constituent tree from a self-attention weight matrix with a greedy or chart-based parsing algorithm.

2.2 Estimating Split Scores Based on the Strength of Inside and Outside Association

In a constituent tree, intuitively the strength of the association among words inside a constituent (inside association) is stronger than that of association to words outside the constituent (outside association). The syntactic distance can be considered to be related with the outside association, which measures the distance between two adjacent spans. However, it does not consider the inside association, which measures the distance between words inside the span.

Based on the aforementioned intuition, we compute the split score according to the inside association together with outside association. We first define the span score which measures how likely a span can be a constituent. For this, we again resort to the self-attention weights of pretrained language models, as they can be regarded as a relatedness metric between words. Given a span (x, y) from position x to y, the stronger the inside association and the weaker the outside association, the more possible that span (x, y) is functioning as a constituent. Therefore we estimate two scores for (x, y): sin(x, y) measuring the strength of the inside association and sout(x, y) for the outside association. We define sin(x, y) as the average attention weights aij inside the span which can be formulated as follows:

Generally, the span score of a span is not only related to its adjacent span, but all other words outside the span. Therefore we formulate the outside association as the average attention weights between the words inside the span and all other words outside the span:

where n is the length of the sentence that consumes the span (x, y). The numerator of the above equation is the sum of all attention weights aij corresponding to the outside association. The first component in the numerator estimates the attention association from words inside the span to words outside the span while the second indicates the attention association from words outside the span to words inside the span.

Given the inside association score sin(x, y) and the outside association score sout(x, y), the score of span (x, y) to be a constituent can be estimated as follows:

Given the span score of two adjacent spans: sspan(x, z - 1) and sspan(z, y), we can estimate the split score as the sum of two span scores:

The estimated split scores could be used for parsing with a greedy or chart-based algorithm.

3 Few-Shot Parsing

The attention weight matrix of BERT is the normalized dot product of a query Q and a key K, as shown in Eq. (6). The essential information of the query and key is from the hidden representation HL-1 of the previous layer. They are computed as the product of HL-1 with WQ and WK, as shown in Eq. (7). We denote the dimension of HL-1 as dmodel. WQ,WK ∈ Rdmodel×dmodel/h , h is the number of attention heads. BERT uses two matrices (WQ,WK) to linearly project HL-1 onto different sub-spaces and then computes the attention matrix as follows:

We have explored all attention weight matrices of BERT to unsupervisedly parse sentences 2 , and have found that the parsing results of different heads vary greatly even in the same layer. The differences among different heads are essentially from the different two linear projections: WQ and WK. Therefore, we conjecture that they have a great impact on parsing and that we may infer better parse trees with better projections.

To verify this, we freeze other parameters of BERT, and retrain the two linear projections (WQ,WK) with annotated trees. Given the binarized tree of a sentence of length n, there are n-1 split points in the tree, and each split point is corresponding to a span of the sentence and an internal node of the tree. We denote the jth split point as splitj and the span split by splitj as spanj . We can compute the split scores of these split points with Eq. (1) or Eq. (5). To make the computation suitable to our training, we normalize the split scores of each span using the softmax function:

2 The experiment results can be found in Section 5.2 of our full paper [Zeng and Xiong, 2022]

where θ is the two matrices WQ, WK, ssplit(spanj , k) is defined in Eq. (1) or Eq. (5). The normalized scores can be seen as the probability of choosing a split point from the corresponding span. Multiplying the probabilities of all split points in a tree, we can get the probability of this tree. Given a binarized ground-truth tree, which contains n - 1 split points, we can compute the probability of the tree as follows:

where splitj ∈ tree denotes that splitj is corresponding to a node in the tree and S represents the sentence. We maximize the probability of ground-truth trees to optimize θ, which is equivalent to minimizing the negative log likelihood:

where N is the number of annotated trees used for training θ, treei is the binarized annotated tree of sentence Si. Since the parameters to be tuned are just WQ and WK, the two linear projection matrices, we can use a few tree samples to well train them. At inference, we use the trained projection matrices to replace the original ones in BERT.

4 Experiments

We evaluated our unsupervised and few-shot parsers on PTB [Dahlmeier et al., 2013] for English and SPMRL [Seddah and others, 2013] for eight languages. For experiments on PTB, our models were built based on the pretrained base-uncased BERT released by Devlin et al. [2019]. While for experiments on SPMRL, our models were built based on the pretrained multilingual-base-uncased BERT released by Devlin et al. [2019]. We denote our unsupervised parser based on the syntactic distance (outside association) as UPOA, the unsupervised parser based on the strength of both the inside and outside association as UPIO, the few-shot parser based on the syntactic distance (outside association) as the FPOA, and the few-shot parser based on both the inside and outside association as the FPIO. The detailed experiment settings can be found in Zeng and Xiong [2022].

4.1 Unsupervised and Few-Shot Parsing on PTB

Unsupervised parsing. We compared UPOA and UPIO with the previous unsupervised parsing methods on PTB. The results are shown in Table 2. Although UPOA and UPIO

underperforms the state of the art (CPCFG), they are substantially better than the other models, especially on short sentences (WSJ10), which indicates that the self-attention weights of BERT can actually approximate the syntactic distances and be used to estimate the association scores.

Few-Shot parsing. We also compared FPOA and FPIO with a few-shot parsing method [Shi et al., 2020] and the supervised state of the art (Berkeley parser [Kitaev et al., 2019]) on PTB. The results are shown in Table 3. Training on a few (10,20,40) annotated trees, both FPIO and FPOA outperform the supervised Berkeley parser substantially and consistently. FPIO is also better than the compared few-shot parser [Shi et al., 2020] training on 10 annotated trees.

4.2 Cross-Lingual Parsing on SPMRL

We evaluated the cross-lingual parsing performance of both unsupervised parsers (UPOA, UPIO) and few-shot parsers (FPOA, FPIO) on SPMRL datasets [Seddah and others, 2013]. We first tuned the hyper-parameters of (UPOA and UPIO) and the parameters of (FPOA and FPIO) on PTB, and then evaluated them on SPMRL dataset [Seddah and others, 2013]. We compared our models with a multilingual unsupervised parser [Kim et al., 2020b], and the multilingual Berkeley parser [Kitaev et al., 2019]. The detailed experiment settings can be found in our full paper [Zeng and Xiong, 2022]. The results are shown in Table 1. Both our unsupervised parsers (UPOA, UPIO) and few-shot parsers (FPOA, FPIO) significantly outperform the supervised Berkeley parser on 5 languages (8 languages in total). UPOA is also superior to the compared unsupervised parser [Kim et al., 2020b] on another 5 languages.

4.3 Analysis

More experiments and analyses can be found in our full paper [Zeng and Xiong, 2022], from which we have the following findings: 1) both UPOA and UPIO only need a few (even only 1) annotated trees for hyper-parameter tuning, 2) FPOA/FPIO achieves the best few-shot parsing performance on middle layers of BERT, 3) the linear projection matrices in FPOA/FPIO are in low-rank, 4) we can detect constituents from sentences by visualizing the attention weight matrices used for parsing.