Function-words Adaptively Enhanced Attention Networks for Few-Shot Inverse Relation Classification

1 Introduction

Relation Classification (RC) aims to classify the relation between two given entities based on their related context. Specifically, given a sentence in a natural language, a set of relation names, and two entities, we want to determine the correct relation between two entities. RC is widely used in natural language processing, such as knowledge base completion [Dong et al., 2020], dialog system[Lee, 2021]. Most existing approaches to RC are based on supervised learning, and the datasets used in training depend on manually labeled data, which limits the classification performance with only a few instances. Therefore, it is necessary to make the RC models able to handle relations with few instances. Due to the success of few-shot learning in computer vision domain [Wu et al., 2021; Yang et al., 2022], Han et al. [2018] first investigated the problem of few-shot relations classification

* These authors contributed equally to this work and should be considered co-first authors.

† Corresponding authors.

(FSRC) and proposed the dataset FewRel1.0 for evaluating the performance of FSRC models. Since then, several other FSRC models have been reported in the literature and they demonstrate remarkable performance on FewRel1.0.[Gao et al., 2019a; Qu et al., 2020; Yang et al., 2021].

However, our experiments show that the performance of existing models for FSRC is significantly reduced when one relation is the inverse of another relation in the given set of relations. Figure 1 shows a few-shot inverse relations classification (FSIRC) task where the relations set contains two relations 'has part' and 'part of '. From the two support instances, we can see that the relation 'has part' is the inverse relation of 'part of '. We note that 'has part' and 'part of ' have the same content word 'part' but different function words 'its' and 'of '. Existing models[Sun et al., 2019; Bao et al. ] for FSRC focus on characterizing the differences between content words but ignore the differences of function words. As a result, these models do not perform well when one relation is the inverse of another relation.

In practical applications, it is often the case that one relation is the inverse of another relation. For example, we found that 21.25% of relations in FewRel1.0 dataset are inverse.

However, it is useful but challenging to classify relations with the presence of inverse relations in few-shot scenarios, as the lack of sufficient samples makes it hard to determine the importance distribution of class-related function words.

In order to address this issue, we propose a new approach to the problem of FSIRC, called Function-words Adaptively Enhanced Attention Networks (FAEA). As shown in Figure 2, in the instance encoder, besides considering the importance of keywords, we use a hybrid attention to capture function words. Specifically, the class-general attention mechanism learns general function words importance distribution. As function words appearing in the same phrase with keywords are more likely to be informative [Zhang et al., 2020], we design a class-specific attention by strengthening function words importance adjacent to keywords. From our experience, in some cases, function words far from keywords are also important. For this reason, we introduce semanticrelated attention for computing the direct semantic relevance between function words and keywords. However, the introduction of function words may increase the intra-class differences. So we present a message passing mechanism to capture and transfer inter-class differences and intra-class commonalities between instances. But when inter-class differences are large, they will bring in noises and thus useful relation semantics can be lost. To avoid this issue, we adaptively control the proportion of transferred inter-class message. Our experiments show that FAEA significantly outperforms major baseline models for FSIRC. Our code is available at https://github.com/DOU123321/FAEA-FSRC.

In a nutshell, our contributions are listed as follows:

• We present FAEA that uses a hybrid attention to capture class-related function words and an adaptive message passing mechanism to reduce intra-class redundancy caused by function words.
• We mathematically show that the involvement of function words will increase intra-class differences from dotproduct measurement and the designed message passing mechanism effectively reduces the redundancy.
• We conduct experiments with two datasets and the results show that our model significantly outperforms the baselines, especially for FSIRC task. Ablation experiments demonstrate the effectiveness of the proposed modules.

2 RelatedWork

Few-shot relation classification predicts novel relations by exploring a small number of labeled instances. Existing methods can be mainly divided into two categories: Gradientbased and metric-based models. A gradient-based method [Finn et al., 2017; Abiola and Andreas, 2019; Qu et al., 2020] can quickly adapt the model to a specific task through a few update steps. MAML[Finn et al., 2017] is a representative model, learning appropriate initialization parameters of the model from base classes and transferring these parameters to novel classes. And metric-based models[Snell et al., 2017; Gao et al., 2019a; Ye and Ling, 2019; Wen et al., 2021] learn the distance distributions among classes, and the same class instances are adjacent in the distance space. As a representative model, Prototypical Network (PN) [Snell et al., 2017] calculates the prototype for each class and classifies query instances by calculating their Euclidean distances. Some models[Ye and Ling, 2019; Wen et al., 2021] add attention mechanisms to enhance PN for highlighting crucial instances and features, but they ignore the intra-instance differences. Some models [Bao et al., 2020; Yang et al., 2021] capture local content words to obtain fine-grained information and ignore function words. However, inverse relations of FSRC has not been effectively handled. This work focuses on inverse relations and proposes a hybrid function words attention to model subtle variations across inverse relations.

3 Our Method

3.1 Problem Statement

FSRC is defined as a task to predict the relation y between the entity pair (h, t) mentioned in a query instance xq , given a support set S and a relation set R, S = {(xik , hik , tik , ri , yi ) , i = 1, . . . , N; k = 1, . . . , K} and R = {y1 , y2 , . . . , yN }, where (xik , hik , tik , ri , yi ) means there is a relation yi between the entity pair (hik , tik ) in the instance xik , and ri is corresponding relation description. N is the number of relations, and each relation with quite small K labeled instances. For a FSIRC task, the relation set R includes some pairs of inverse relations. For example, 'participant' and 'participant of ' are inverse relations, and their relation descriptions are "person that actively takes part in the event" and "event a person was a participant in", respectively.

3.2 Overall Framework

As shown in Figure 2, our model consists of three parts:

• InstanceEncoder. Given an instance and entity pair, we employ instance-level global encoder and phrase-level local encoder to encode the instance into an embedding.
• Function-words Enhanced Attention. Phrase-level local encoder utilizes function-words enhanced attention to capture important function words in instances.
• Adaptive Message Passing. After computing embeddings, we transfer commonalities between same class instances and differences between different class instances.

3.3 Instance Encoder

Given an instance x = {w1, . . . , wl} mentioning two entities with l words, we use BERT [Devlin et al., 2019] as the encoder to get corresponding embeddings X = {w1, . . . ,wl}, where each word embedding wi ∈ Rd and d is embedding dimension. For i-th relation ri , we encode the name and description to get relation word embeddings Ri ∈ Rl×d , and use hidden states of [CLS] token to obtain features of relations ri ∈ R2d .

For instance xik in S and query instance xq , our model generates global instance embeddings and local phrase embeddings to form hybrid instance embeddings xik and xq . The following takes xik as an example to explain.

Instance-level Global Encoder. The global features {xik ∈ R2d , i = 1, . . . ,N; k = 1, . . . ,K} are obtained by concatenating corresponding hidden states of two entity mentions according to [Soares et al., 2019].

Phrase-level Local Encoder. The main process consists of learning a keyword attention αik ∈ Rl and a function-words attention βs ∈ Rl . αik can be computed as follows:

where the memory unit uw ∈ Rd is a trainable parameter. It can help us to select general keywords from instances. Then we obtain βs according to 3.4 and form phrase-level local features ˜xik , computed as follows:

In short, we additionally attend local phrase-level information to learn the subtle differences of inverse relations.

3.4 Function-words Enhanced Attention

We utilize class-general attention to learn general function words importance distribution and leverage class-specific attention consisting of constituent attention and semanticrelated attention to estimate class-specific importance.

Class-general Attention. By downweighing the importance of words related to uw and upweighing the words importance irrelated to uw, we get general function-words importance βgeneral ∈ Rl , where E ∈ Rl is an all-one vector:

Class-specific Attention. Considering function words importance varying by class, we learn a constituent prior matrix C ∈ Rl×l and a semantic-related matrix S ∈ Rl×l to strengthen the attention of function words adjacent to keywords.

The element Ci,j means the probability that wi and wj in instance xik belong to the same phrase, obtained as follows, where [·]n is the n-th row of a matrix.

We compute the score sn,n+1 representing the tendency that wn links to right neighbor wn+1. Then, constrain wn to either link to its right neighbor or left neighbor. This constraint is implemented by applying a softmax function to two attention links of wn. As pn,n+1 and pn+1,n may have different values, we average its two attention links.

We also use self-attention mechanism to obtain matrix S to attend necessary function words far from keywords:

Next, we find the keywords index Iik in xik , computed as:

where max(·)r is used to get indexes of the top-r largest attention keywords, and r is the number of keywords. And then, we strengthen related function words according to C and S:

Finally, the model uses βconstituent , βrelated and βgeneral to form hybrid function-words attention vector β, formalized as:

where λ is hyper-parameter.

All in all, inspired by MAML[Finn et al., 2017] learning general model parameters and fine-tunning them to adapt to the specific task, we design class-general and class-specific attention to learn function-words variance in few-shot setting.

3.5 Adaptive Message Passing

Adaptive Message Passing reduces intra-class redundancy caused by function words and adaptively controls the proportion of transferred inter-class message.

Firstly, we construct a directed graph G = (V, E), where V = [x11 ; . . . ; xKN ] is a set of instances features with |V | = N ×K and E is the adjacency matrix. [; ] denotes the rowwise concatenation and vi ∈ Rd denotes the i-th row of matrix V.

We design a new node updating way that captures and transfers inter-class differences and intra-class commonalities between instance nodes, according to the work of [Bo et al., 2021]

where N 0 i and N 1 i denote the different and same class neighbor set as vi , respectively. di is the degree of node i.

For i-th relation, we average K supporting features to form prototype representation pi following [Snell et al., 2017].

With N prototype representations, we calculates the probability that query instance xq belongs to the relation i:

The final objective function is formally written as:

where y is relation label, and zy is estimated probability for the relation y.

In short, we design a new node updating method to capture inter-class differences in few-shot setting.

3.6 Theoretical Analysis

In this section, we theoretically prove that the involvement of function words increases intra-class differences (Theorem1) and the designed message passing mechanism makes different class nodes become discriminative and same class nodes similar (Theorem2).

Given any two instances xi and xj, let the corresponding keywords representations be xci = {ai1, ai2, . . . , aid} ∈ Rd and xcj = {bj1, bj2, . . . , bjd} ∈ Rd , the function words representations be xfi = {a1i , a2i , . . . , adi } ∈ Rd and xfj = {b1j , b2j , . . . , bdj } ∈ Rd , the instance representations considering function words be xi = xci + xfi and xj = xcj + xfj .

where norm(xi ⊙ xj) is computed as follows:

and ⊙ indicates the inner product of vectors.

Theorem 2. Given any two instances considering function words xi and xj, define the similarity measure between xi and xj as

The message passings between same class instances and different classes are respectively defined as follows:

and

· If xi and xi belong to the same class, then,

• If xi and xi belong to different classes, then,

4 Experiments

4.1 Baselines

• To demonstrate the usefulness of local-level features, compare with global-level models: metric-based models Proto [Snell et al., 2017], Proto-HATT [Gao et al., 2019a], MLMAN [Ye and Ling, 2019], BERT-PAIR [Gao et al., 2019b] and TPN [Wen et al., 2021]. And gradient-based models MAML [Finn et al., 2017] and GNN [Satorras and Estrach, 2018].
• To prove function words importance, compare with word-level models: TD-Proto [Sun et al., 2019], using memory network to learn content words importance. ConceptFERE [Yang et al., 2021], designing attention mechanism to measure class-specific words importance.
• Models introducing external information: REGRAB [Qu et al., 2020] utilizes relation graph knowledge. CTEG [Wang et al., 2020] uses dependency trees.
• Pretrained RC models: MTB [Soares et al., 2019] pretrained with matching the blank task. CP [Peng et al., 2020], a framework pretrained with entity masked.

4.2 Datasets and Settings

We evaluate our model on FewRel1.0 [Han et al., 2018] and FewRel2.0 [Gao et al., 2019b] in terms of the accuracy under multiple N-way-K-shot tasks. And we select N to be 5 and 10, K to be 1 and 5 to form 4 test scenarios according to [Gao et al., 2019a]. In addition, we take base-uncased BERT as the encoder of 768 dimensions for a fair comparison. The input max length is set to 128. Besides, the AdamW optimizer is applied with the learning rate as 2 × 10-5 and weight decay as 1×10-2 . Furthermore, hyper-parameter λ is set to 0.6 and uw is randomly initialized following [Sun et al., 2019].

4.3 Results

Performance on FSRC. As shown in Table 1, our method is significantly better than the strong baselines, especially under 1-shot settings. Specifically, our model improves 5-way1-shot and 10-way-1-shot tasks by 4.80 points and 6.06 points in terms of accuracy, demonstrating superior ability. In addition, our method achieves good performance on FewRel2.0, as shown in Table 2.

• Proto and GNN, as widely-used baselines for few-shot learning, perform not well on FSRC. Unlike low-level patterns can be shared across tasks in computer vision, words that are informative for one task may not be relevant for other tasks. But these models ignore such local words importance variations in learning. But FAEA leverages phrase-level attention to attend local features.
• TD-PROTO and ConceptFERE also use semantic-level attention to explore content words, but neglect function words maintaining syntactic structure differences. Since FAEA captures function words to form fine-grained features, it obtains better performance.
• When computing relation prototypes, Proto-HATT and TPN utilize intra-class commonalities, not considering inter-class differences. FAEA captures and leverages differences to get more discriminative representations.

Performance on FSIRC. To further illustrate themodel effectiveness for FSIRC, we evaluate models on FewRel1.0 validation set with different scenarios, shown in Table 3. Randommeans general evaluation setting, that randomly samples 10,000 FSRC tasks from validation set. Inverse represents each evaluated task including inverse relations. The baselines perform well under random scenarios but drop significantly under inverse settings, around 26.98 points in 1-shot scenarios, which illustrates that FSIRC is extremely challenging. FAEA achieves the best accuracy, especially under inverse settings, proving that it can effectively capture function words and handle FSIRC tasks.

5 Analysis

5.1 Analysis of FunctionWords Attention

This section discusses the effect of function-words attention. As shown in Table 4, removing phrase-level (Model 2) and function-words attention (Model 3) severely decreases the performance, indicating function words are also essential to

represent relations. Furthermore, as shown in Figure 4, with

the help of function-words attention, we highlight 'are' and

'of ' to form 'are part of ', which appears in query and sup-

port instance of class 'part of ', then this support instance gets

a higher score, and our model correctly classifies the query.

score

Figure 4: A 2-way-1-shot FSIRC task and visualizes attention scores of words by FAEA.

To demonstrate the effectiveness of three components of function-words attention, from model 4,5,6 of Table 4, we can see a performance decline if three components are removed separately. As shown in Figure 5, TD-Proto mainly attends content words such as 'parts', 'Towada' and 'Lake'. FAEA without general attention enhances not only function words importance but also content words irrelated to keywords, such as 'mountains' and 'Hakkodda'. FAEA without constituent attention enhances some keywords-irrelated function words such as 'along', 'and'. FAEA without related attention tends to decrease some related function words importance far away from the keywords, such as 'are'. FAEA further captures correct function words to form the phrase 'are three parts of ' , demonstrating that three components all contribute to enhancing function words importance.

5.2 Analysis of Adaptive Message Passing

As shown in Table 4, we compare models without message passing (Model 7) and message passing without mean (Model 8). We observe that considering message passing achieves higher accuracy, and adding the mean to control the proportion of transferred message further improves the performance.

To further demonstrate the effectiveness of message passing, we choose some classes and visualize the similarity shown in Figure 3. We can see that only considering content words, the inter-class information of inverse relations has a high similarity score. And the introduction of function words effectively reduces it. But from the left part, function words reduce the similarity of intra-class information. The designed message passing without mean effectively increases intra-class commonalities and keeps inter-class differences. But from the right part, when the inter-class differences are large enough, with the message passing mechanism, the similarity of the inter-class sharp decline, and it will destroy the original relation semantic.

6 Conclusion

In this paper, we have presented FAEA, a framework that can effectively handle few-shot inverse relations by enhancing related function words importance. Experiments demonstrate that FAEA achieves new SOTA results on two NLP tasks on FewRel dataset. In future work, we will try to design a more effective and general function-words enhanced backbone network for various NLP tasks.

Acknowledgments

This work was supported by National Key R&D Program of China (2019YFB2102404), National Natural Science Foundation of China (NSFC) (61972455), National Natural Science Foundation of China (NSFC) (61976153), the Joint Project of AISHU.com and Bayescom. Xiaowang Zhang is supported by the program of Peiyang Young Scholars in Tianjin University (2019XRX-0032).