Learning Few-shot Sample-set Operations for Noisy Multi-label Aspect Category Detection

1 Introduction

Multi-label Aspect Category Detection (MACD) [Tulkens et al., 2020] is a crucial task for aspect-based sentiment analysis [Pontiki et al., 2016], which aims to identify aspect categories in a given sentence. Generally, a sentence contains more than one aspect category, i.e., it can be viewed as a multi-label classification problem. Last few years, MACD received widespread attention. However, most existing methods [Li et al., 2020] heavily rely on a considerable amount of labeled data during training. Therefore, their performances can drop dramatically when only a few labeled samples exist for some aspect categories. Intuitively, few-shot learning is of great significance in solving MACD.

∗Corresponding author

Support Set

(A) Service

1. Nice service but food cost a lot.
2. You should ask server well in advance about the internet connection.

(B) Money

1. There was a charge for wifi.
2. High rates for just ok room service but the internet seems to only work at times.

(C) Internet

1. There was a charge for wifi.
2. You should ask server well in advance about the internet connection.

Query Set

• (A), (B), (C) (1) I access the ethernet cable at the corner due free service.

(B), (C)

(A)

1. The hotel is everything I expected:great price, good breakfast, free wifi.
2. Our server thomas made the experience that much more enjoyable.

Table 1: A 3-way 2-shot example. The colored boxes indicate target aspects, while the gray boxes highlight irrelevant aspects.

However, most few-shot learning methods (e.g., prototypical network) focus on single-label prediction, i.e., each sentence is restricted to one label. Therefore, they are volatile for the MACD task in the noisy scenario. The main challenges are summarized as follows: (1) Each class prototype is closely related to the target aspects of intra-class support samples, whereas irrelevant aspects interfere with prototype extraction. For example, in Table 1, for the class "Service", its corresponding prototype may receive negative impacts from irrelevant aspects "food", "cost a lot", and "internet". (2) Under the multi-label setting, some support samples are shared among classes, failing to distinguish the class prototypes. Table 1 shows that the sample "There was a charge for wifi" is shared by the classes "Money" and "Internet". For these two classes, the prototypes may be indistinguishable due to the high-similar sample distribution. Inevitably, the above issues complicate class prototype extraction. (3) The number of aspect categories in a sentence is intangible for multi-label prediction. Therefore, a low-cost and high-efficiency method is expected to address the noisy MACD task.

Existing works focus on contrastive learning paradigms or attention networks to alleviate noise interference for the MACD task. However, merely learning prototypes by training contrastive objects [Liu et al., 2022; Zhao et al., 2022] or attention weights [Hu et al., 2021] fails to fully address the noise caused by the irrelevant aspects and similar sample distribution. Therefore, they struggle to handle the noisy MACD task in fewer sample scenarios. Shortly, their performances drop significantly when each class only has two or three training samples. Therefore, it remains a considerable challenge for the noisy MACD task.

To solve the above issues, we revisit the MACD task from a new perspective and propose a group of Few-shot Sample-set Operations (FSO) to handle the noisy MACD task by identifying the semantic contents of samples. The set operations are realized as the arithmetic operations on samples and gain considerable progress in image synthesis [Alfassy et al., 2019]. Therefore, we apply the concept of set operations to the prototypical network and design FSO to handle noisy MACD in fewer sample scenarios. The FSO imitates arithmetic operations through intersection (Mint), subtraction (Msub), and union (Muni) networks. The Mint receives two samples and produces a feature vector with their common semantic content, excluding the other aspects in the original samples. Inversely, the feature vector generated by Msub removes the shared content and reserves the irrelevant aspects. Besides, theMuni is implemented on the outputs of Mint andMsub to restore the original samples. Learning the interactions among the Mint, Msub, and Muni, the FSO analyzes the semantic contents of the sample and distinguishes irrelevant aspects from it. For support set, the FSO utilizes the shared features within a class to extract discriminative prototypes. For query set, the FSO takes category description as prior knowledge to customize a dedicated query vector for each class. To meet actual practice, we apply the FSO to a multi-label architecture. And score-wise loss and multi-label loss are implemented on the architecture to promote the learning of the FSO and throw the trouble of threshold setting. The contributions are summarized as follows:

• We propose the FSO to solve the noisy MACD task by distinguishing the semantic contents of samples. Learning the interactions among Mint, Msub, and Muni, our method alleviates the noises to generate discriminative prototypes for support set and dedicated query vectors for query set to estimate label-sample relevance. And the FSO works well with fewer sample scenarios.
• We design a multi-label architecture that integrates with score-wise loss and multi-label loss to optimize the FSO for multi-label prediction, avoiding complex threshold training and selection.
• Extensive experiments show that our method outperforms strong baselines. Besides, the method is not limited to the MACD task. It can also be applied to more complex tasks, e.g., sentence embedding representation, since it better separates embedding features than conventional contrastive learning in fewer sample scenarios.

2 RelatedWork

2.1 Multi-label Aspect Category Detection

MACD is a subtask of aspect-based sentiment analysis, which aims to identify aspect categories from a predefined set. The previous works can be summarized into two groups: supervised and unsupervised methods. Supervised methods [Schmitt et al., 2018; Cai et al., 2020] heavily rely on a large amount of labeled data to learn features for each aspect category. Therefore, they suffer from the long-tail distribution [Yu et al., 2021] for some aspect categories with a few labeled data. Unsupervised methods [Tulkens et al., 2020] are poorly performed by mining aspect knowledge in massive unstructured texts. Therefore, recent works keep an eye on few-shot learning.

2.2 Multi-label Few-shot Learning

The meta-learning [Hospedales et al., 2021] is a mainstream few-shot learning line, including model-based [Tsendsuren and Hong, 2017], optimization-based [Lee et al., 2019], and metric-based [Sung et al., 2018; Assran et al., 2022; Wang et al., 2021; Lv et al., 2021] methods. However, most of them only work well in the single-label setting and fail to address high-noise multi-label tasks. To the best of our knowledge, the research works on multi-label few-shot learning mostly focus on contrastive learning paradigms and attention networks in the text domain. Yang et al. [2020] utilize contrastive learning to push positive and negative samples away from each other. However, they have limited performances due to the neglect of adverse effects produced by irrelevant aspects. Then, Hu et al. [2021] and Yan et al. [2022] leverage attention networks to alleviate the noise from irrelevant aspects. However, they are inefficient when many highsimilar samples exist in different classes. From a new perspective, we design the FSO to solve the above issues. The concept of set operations is proposed by Alfassy et al [Alfassy et al., 2019], and they use label-set operations to generate more data on image synthesis. Compared with them, we design novel sample-set operations without data generation and introduce score-wise loss and multi-label loss to improve performances on the MACD task.

3 Methodology

3.1 ProblemFormulation

We follow the episodic paradigm to train a meta-learner for the noisy MACD task. In the label space, the data can be divided into Ctrain (known) and Ctest (unknown), where Ctrain ∩ Ctest = ∅. And a meta-task includes support set S and query set Q. In the meta-train, N unique classes are sampled from Ctrain, and then K samples are sampled from each class to construct S (i.e., Nway K-shot formulation), which can be denoted as S = {(x11 , y1 1 ), ..., (xK1 , yK 1 ), ..., (xKN , yK N )}. And Q includes T samples sampled from the remaining samples of the same N classes, i.e., Q = {(x11 , y1 1 ), ..., (xT1 , yT 1 ), ..., (xTN , yTN )}. In the meta-test, we need to construct the support set and query set from Ctest. The meta-learner aims to predict the class label of the query set based on the support set.

3.2 Architecture Overview

The proposed architecture is shown in Figure 1. Learning the interactions among the Mint, Msub, and Muni, the FSO imitates arithmetic operations to analyze the semantic contents of samples. We first introduce the learning of the FSO and then apply it to support and query sets for multi-label prediction.

3.3 Few-shot Sample-set Operations

The FSO includes three operations, i.e., Mint, Msub, and Muni. These three operations refer to mathematical symbols (e.g., ∩, -, and ∪). Given two feature vectors X and Y , the Mint extracts the shared features between them, the Msub identifies the exclusive features of X excluding Y , and the Muni denotes the merged features from them. To ensure effects, the Mint, Msub, and Muni networks map the synthesized feature vectors of X and Y to feature space F:

The Mint, Msub, and Muni follow the same network structure and different arithmetic units (e.g., min(.), sub(.), and

where Relu(.) is activation function. [X; Y ] denotes the concatenation between X and Y . W1, W2, W3, b1, b2, and b3 are learnable parameters. The AU(.) denotes the arithmetic unit.

Score-wise Loss

The score-wise loss is used to ensure that the Mint, Msub, and Muni can capture correct semantic features on samples. Specifically, the score-wise loss is based on cosine similarity scores. And the cosine similarity [Yan et al., 2022] is widely adopted in the prototypical network to measure the similarity scores between feature vectors in the feature space F:

where XT is transpose of X and ||.|| is L2-norm of vectors.

The cosine similarity scores range between -1 to 1. Therefore, we use the highest score (i.e., 1) to design the score-wise loss LFSO. The FSO utilizes the score-wise loss LFSO to enforce the capacity to capture semantic features on samples. The LFSO includes Lsym FSO and Lnorm FSO to ensure the symmetry and normalization of the Mint, Msub, and Muni:

The following loss Lsym FSO is used to rectify symmetric Mint andMuni:

where F XY ∗ and F YX ∗ derive fromM∗(.) with reversed order of the inputs, and ∗ ∈ (uni, int).

The second loss Lnorm FSO is realized as a criterion to normalize the Mint, Msub, and Muni:

3.4 Support-set Processing (SP)

Each class in the support set includes K samples, describing the common aspect category. However, the noises complicate the prototype extraction. To alleviate noise interference, we take category description (i.e., label description) as prior knowledge to learn sample representations with category features. Then, the class prototypes are extracted by the FSO through the shared features within a class to further eliminate the negative effect caused by noises.

Support Sample Feature Extraction

Given a support sample s = {s1, s2, ..., sn} of n tokens and its category description c = {c1, ..., cm} of m tokens, we use special tokens [CLS] and [SEP] to concatenate the sample and its category description. The "[CLS], s, [SEP ], c, [SEP ]" as input is converted to the encoder (e.g., BERT [Devlin et al., 2019]) to generate hidden states Hs:

where hs ∈ Rn∗d denotes the hidden states of the support sample with category features, and hc ∈ Rm∗d indicates the hidden states of the category description. And d is the hidden dimension. Therefore, we define hji s as the hidden states of j th support sample in ith class.

The sample feature vector f j i and the corresponding category feature vector gj i for j th sample in ith class are obtained through a mean pooling layer:

where f j i ∈ R1∗d , gj i ∈ R1∗d , hji s ∈ Rn∗d , and hji c ∈ Rm∗d . For ith class, there are K category feature vectors (i.e., g1 i , g2 i , ..., gK i ), and they express the common class information. We assign importance weights to them to map a final category feature vector gi:

where A1 is the weight matrix. g′i = [g1 i , g2 i , ..., gK i ] ∈ RK∗d and gi ∈ R1∗d . W4 and W5 are learnable parameters.

After processing the support set, we obtain N ∗ K sample feature vectors {f 1 1 , ..., f K 1 , ..., f 1N , ..., f K N } and N category feature vectors {g1 , g2 , ..., gN }.

Class Prototype Generation

In the same class, all samples express the common aspect content. The FSO utilizes the shared content within a class to extract the corresponding prototype. Specifically, the FSO receives the sample feature vectors [f 1 i , f 2 i , ..., f K i ] from ith class. And then, it outputs the intersection features as the common aspect features from the same class by learning the interaction among Mint, Msub, and Muni networks:

where F i int ∈ Rr∗d . And r is the number of intersection feature vectors extracted by the FSO for ith class.

The discriminative prototype pi for ith class can be extracted by assigning importance weights to the F i int :

where A2 is the weight matrix. pi ∈ R1∗d . W6 and W7 are learnable parameters. The SP component can extract N class prototypes, i.e., pi, i ∈ {1, 2, ...,N}.

3.5 Query-set Processing (QP)

The query sample may contain more than one target aspect category in noisy scenarios. Therefore, it is non-trivial to customize a dedicated query vector for each class.

Query Sample Feature Extraction

Given a query sample q = {q1, q2, ..., qt} of t tokens, we feed "[CLS], q, [SEP]" into the encoder to get hidden states Hq:

where hq ∈ Rt∗d is the hidden states of the query sample.

The query-generic feature vector ¯u is obtained through a mean pooling layer.

where ¯u ∈ R1∗d .

Dedicated Query Feature Learning

The FSO takes category description as prior knowledge and extracts target-related features to customize a dedicated query vector for each class. Specifically, we feed the query-generic feature vector ¯u and the corresponding category feature vector gi into the FSO, then it outputs the intersection feature as the query-special feature:

where ui ∈ R1∗d . The query-special feature is realized as a dedicated query feature vector for each class. Therefore, the QP component can obtain N dedicated query feature vectors, i.e., ui, i ∈ {1, 2, ...,N}.

Distance Metric

Given a query sample, the distance similarities between it and class prototypes are defined as Z = {z1, z2, ..., zN} ∈ RN :

3.6 Multi-label Inference

In multi-label inference, some works [Hu et al., 2021; Liu et al., 2022] train a policy network or multi-layer perception to learn a threshold to determine the number of aspects. However, they may fail to get an acceptable threshold to meet the requirement of complicated multi-label prediction. Therefore, we introduce a multi-label loss to optimize training objectives and avoid complex threshold training or selection.

The label of the query sample is y = {y1, y2, ..., yN} ∈ RN , where yi ∈ {1, 0}, and yi = 1 indicates it belongs to ith class. The positive score set Γ = {zi ∈ Z|yi = 1}, and the negative score set Λ = {zi ∈ Z|yi = 0}.

sentences due to random sampling. These two datasets include 100 aspect categories, and we split the 100 aspect categories into 64, 16, and 20 for training, validation, and testing. The detailed statistics are presented in Table 4.

Implementation Details. The proposed method is implemented with PyTorch (version 1.10.0). The uncased English version of BERT is our encoder. Besides, the first six layers of BERT are frozen to reduce the trainable parameters. We conduct experiments on a single GPU (RTX 3090 Ti) with CUDA version 11.3. The model is trained by the AdamW optimizer. For the learning rate, we set 5e-4 in the FSO and 1e-4 in other network structures. Meanwhile, we use the GradualWarmupScheduler to optimize the learning rate. And we fix the hyper-parameters σ, α, and β as 0.03, 0.3, and 0.7. (Note: the parameters are adjustable). We randomly sample 100 meta-tasks for training and 600 meta-tasks for validation and testing in every epoch.

Evaluation Metric. We follow Zhao et al. [2022] to use Area Under Curve (AUC) and Macro-F1 score for performance evaluation and comparison.

Multi-label Loss

The multi-label loss based on circle loss [Sun et al., 2020] is written as follows:

where t is the threshold, and σ is the temperature scale parameter. The optimal goal of the LMUL is that the target scores are greater than t and the non-target scores are less than t. Directly, we set threshold t = 0 and filter positive scores as the multi-label prediction:

Overall Training Objectives

where α and β are hyper-parameters.

4 Experiments

4.1 Experimental Setup

Datasets. Extensive experiments are conducted on datasets FewAsp (multi) and FewAsp (random). These two datasets originate from a large-scale multi-domain dataset (i.e., Yelp aspect [Bauman et al., 2017]) for aspect recommendation. And FewAsp (multi) consists of multi-aspect sentences, whereas FewAsp (random) contains single- and multi-aspect

4.2 Overall Performance

Following the previous works [Liu et al., 2022], the number of query samples is fixed at 5, and the experiments are conducted on "N = 5, 10 and K = 5, 10" (i.e., N-way Kshot formulation) to compare the performance with the strong baselines. Then, we set "N = 5, 10, 15, 20 and K = 2, 3" to further analyze the performance in fewer sample scenarios. The results are presented in Tables 2, 3, 5, and 6, with the following observations.

(1) Overall, the proposed method outperforms most baselines. The results demonstrate the effectiveness of our method on the noisy MACD task. Though the strong baseline (i.e., LPN [Zhao et al., 2022]) achieves competitive Macro-F1 or AUC for the 10-shot scenario in Tables 2 and 3, the number looks large and takes high training time, which limits the applications in practice. The proposed method achieves significant performance in fewer sample scenarios. Specifically, our method improves upon the most competitive baseline by 1.06%-2.61% AUC and 4.03%-10.30% Macro-F in Table 5. Besides, it obtains 1.27%-2.42% AUC and 9.26%-11.01% Macro-F improvements in Table 6. The reason is that the contrastive learning and attention networks are weak in fewer sample scenarios (i.e., 2-shot or 3-shot). We can solve the noisy MACD task in fewer training samples by distinguishing the semantic contents of samples. In short, all experimental results on benchmark datasets show that our method achieves considerable performance.

(2) The results on FewAsp (multi) are inferior to those on FewAsp (random). FewAsp (multi) presents a more challenging scenario than FewAsp (random) because it includes masses of multi-aspect sentences. In most cases, the proposed method still obtains remarkable improvement over other methods on FewAsp (multi), esp. fewer sample scenarios. Compared with the best baseline, we achieve an average of 1.84% AUC improvement on 2-shot and 3-shot settings. Generally, more classes contain more noise. For 15 and 20 classes, the results surpass strong baselines by 2.42% at most, with an average of 1.79% AUC. Besides, we improve the performance better on the "20-way 3-shot" than on the "5-way

3-shot". The fact indicates that the proposed method can alleviate the noises to address the noisy MACD task.

(3) Under the Macro-F metric, we obtain at most 11.01% and an average of 8.59% Macro-F improvements when there are two or three samples. Therefore, the proposed method is superior to the strong baselines LPN [Zhao et al., 2022] and LDF [Liu et al., 2022] for multi-label prediction. To achieve multi-label results, LPN trains an adaptive multi-label module to determine the threshold, and LDF utilizes empirical knowledge to make threshold selections. However, these methods are unfaithful in fewer sample scenarios. The proposed method can address the noisy MACD task in fewer sample scenarios, avoiding complex threshold training or selection. In conclusion, all experimental results verify the effectiveness of our method.

4.3 Ablation Study

We conduct an ablation study on the 5-way 2-shot scenario from FewAsp (multi). A set of ablation experiments are implemented to examine the structure design of the proposed method. The detailed results are depicted in Table 7. First, the performance drops by 2.04% AUC and 6.22% Macro-F1 when we remove the FSO. The result indicates that the FSO positively solves the noisy MACD task by learning the interactions among the Mint, Msub, and Muni networks. Then, we further implement the ablation study on the FSO. Without Msub or Muni network, the performance of the method is reduced. Besides, the performance is worse when we remove the Msub and Muni networks at the same time. These negative results denote that the interactions among these three networks are of the essence to solving the MACD task. Therefore, any absence of the FSO can decrease our performance. In conclusion, the design of our method is reasonable and achieves the best performance.

4.4 Visualizations

To further analyze the performance, we visualize the embedding representations of prototypes in the feature space compared with dominant LDF and LPN. Specifically, we sample 3000 episodes in the testing set of FewAsp (multi) and visualize the prototype per class by using the visual tool TSNE [Laurens and Hinton, 2008]. The prototypes are generated by intra-class sentences and are closely related to the target aspects. Visualizing the embedding representations of prototypes, we leverage colored spots to observe the prototype distribution. To fair comparison, we set the seed is 15 and perplexity is 30 for all methods. The results are presented in Figure 2 (i.e., 5-way 2-shot) and Figure 3 (i.e., 10-way 2shot). For 20 classes from the testing set, we clearly separate the prototype embeddings in the feature space. The LDF has many spots scattered around clusters, whereas the LPN includes many fuzzy clusters. Compared with LDF and LPN, the results denote that our method can eliminate the negative effect caused by noises to extract representative prototypes.

5 Conclusion

We propose a simple yet effective FSO method to solve the noisy MACD task by distinguishing the semantic contents of samples. Learning the interactions among the Mint, Msub, and Muni networks, the FSO imitates arithmetic operations on samples to distinguish relevant and irrelevant aspects, which aims to automatically analyze the semantic contents of samples. Eliminating the negative effect caused by noises, the FSO extracts discriminative prototypes and customizes the corresponding dedicated query vector for each class. Meanwhile, we design a multi-label architecture integrated with score-wise loss and multi-label loss to optimize the FSO for multi-label prediction, avoiding complex threshold training or selection. Extensive experiments on benchmark datasets show that the proposed method obtains convincing improvements on the noisy MACD task, esp. fewer sample scenarios.

Acknowledgements

This work is supported by the National Key Research and Development Program of China (No. 2021ZD0111202), the Science and Technology Project of State Grid Corporation of China (No. 5700-202341285A-1-1-ZN), the National Natural Science Foundation of China (No.62176005), PKU-Haier "Smart Home" Qingdao Joint Innovation Lab.