Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching

1 Introduction

As an important fine-grained task in multimodal sentiment analysis, Targeted Multimodal Sentiment Classification (TMSC, a.k.a aspect-based multimodal sentiment classification) has received increasing attention in recent years. Given a pair of sentence and image, the goal of TMSC is to identify the sentiment polarities towards each opinion target in the sentence [Xu et al., 2019b; Yu et al., 2019]. For example, in Fig. 1, given the multimodal tweet and its two opinion targets "Nancy Ajram" and "Salalah Tourism Festival", it is expected to identify that the user expresses Positive and Neutral sentiments towards them, respectively.

In the literature, a myriad of deep learning approaches have been proposed for the TMSC task. [Xu et al., 2019b] and [Yu et al., 2020] focused on designing effective attention mechanisms to model the interactions among the target, text, and Figure 1: An Example of Targeted Multimodal Sentiment Classification (TMSC). Nancy Ajram and Salalah Tourism Festival are two mentioned targets. Box-1 denotes the 1st bounding box is aligned with Nancy Ajram, and Box-N/A denotes no bounding box is aligned with Salalah Tourism Festival.

∗Equal contribution.

†Corresponding author.

1 The source code is released at https://github.com/NUSTM/ITM.

image. [Yu et al., 2019] and [Wang et al., 2021] followed the recent pre-train and fine-tune paradigm, and adapted existing pre-trained models to capture the text-image, target-text, and target-image interactions. More recently, [Khan et al., 2021] proposed a Transformer-based image captioning model to translate the image to an auxiliary sentence, and then combined the original and auxiliary sentences for targeted sentiment classification.

However, all these existing studies failed to explicitly consider the matching relations between the target and the image, which is essential for the TMSC task for following reasons:

• Coarse-Grained Image-Target Matching. Based on our observations of a benchmark Twitter dataset of TMSC, around 58% of the input targets are not presented in associated images in a benchmark dataset, and these unrelated images will inevitably bring much noise for the TMSC task. For example, in Fig. 1, given Salalah Tourism Festival as the input target, the unrelated image may mislead the model to predict its sentiment as Positive. Hence, it is crucial to capture the image-target relevance to alleviate the visual noise for targeted sentiment classification.
• Fine-Grained Image-Target Matching. For those targetrelated images, as each image contains a number of visual objects (i.e., fine-grained image), identifying the aligned visual object to the input target is generally helpful for pre-

dicting its sentiment. For example, in Fig. 1, among all the marked bounding boxes, the bounding box with the pleasant woman (i.e., Box-1) provides the most important clue for detecting the Positive sentiment over Nancy Ajram.

Motivated by these observations, we propose a coarse-tofine grained Image-Target Matching network (ITM) for the TMSC task. Specifically, we first construct an Image-Target Matching dataset by manually annotating 1) the relevance between the image and the target and 2) the visual object (i.e., bounding box) aligned with the input target. With such an annotated dataset, we propose a multi-task learning architecture ITM to jointly perform coarse-to-fine grained imagetarget matching and targeted sentiment classification. ITM contains three key modules: the first module is to identify image-target relevance for dynamically controlling the contribution of visual information; with the filtered visual information, the second module focuses on object-target alignment to learn appropriate weights of each visual object based on their alignment probabilities with the input target; the last module performs multimodal fusion and sentiment classification.

Experimental results on two benchmark datasets for the TMSC task show that our multi-task learning model ITM consistently outperforms a number of state-of-the-art methods, and presents insightful and interpretable visualizations, demonstrating the importance of coarse-grained and finegrained image-target matching to the TMSC task.

2 Task Formulation

Given a multimodal corpus D, let us first use {X1,X2, . . . ,X|D|} to denote a set of samples in the corpus. For each sample, we are given an n-word sentence S = (w1,w2, ...,wn), an image V, and an m-word opinion target T = (t1, t2, ..., tm), where T is a sub-sequence of S.We then formulate the three tasks in our work as follows:

Image-Target Relevance. For each sample X = (S,V, T), the target T is assumed to be associated with a relevance label r indicating whether the image V is related to T, where r is either Related or Unrelated. The goal of this task is to learn a binary classification function that maps X to r.

Object-Target Alignment. For each sampleX = (S,V, T), an object detection method is employed to identify K object proposals in the image V, and the target T is associated with its alignment distribution over the K object proposals, denoted by A. The goal of this task is to learn a mapping from X to the alignment distribution A.

TMSC. For each sample X = (S,V, T), we assume that the target T is associated with a sentiment label y, which can be Positive, Negative or Neutral. The goal of this main task is to learn a sentiment classifier that maps X to y.

3 Dataset

We construct an Image-Target Matching dataset for ImageTarget Relevance and Object-Target Alignment tasks.

Source. Since both tasks require the annotation of targets, we construct our dataset based on a subset of one benchmark dataset for the TMSC task (i.e., TWITTER-17), which has annotated the targets by [Lu et al., 2018]. We randomly select 1176, 588, and 588 samples from the training, development, and test sets of TWITTER-17, and employ two PhD students for annotation. The annotation for the Image-Target Relevance task reaches an agreement of 98.5%, and the agreement for bounding box annotation is 92.3%, indicating the high quality of our data. For disagreement samples, we ask a third expert to make the final decision.

Statistics and Analysis. The basic statistic of our dataset is shown in Table 1. It can be seen that a large percentage of targets are unrelated to images. For each target-related image, since the semantic meaning of the target is clear, only one bounding box is annotated. Fig. 2 (left) shows the distribution of bounding box area over image area ratio. Compared to images, most bounding boxes are relatively small, which implies the challenge of object-target alignment. In Fig. 2 (right), we further show the correlation between sentiment and image-target relevance. It is interesting to observe that for targets related to the images, users tend to express either positive or negative sentiment towards them; whereas for targets unrelated to the images, users tend to express neutral sentiment over them. This indicates image-target relevance indeed provides important clues to TMSC.

4 Methodology

We propose a multi-task learning framework named coarseto-fine grained Image-Target Matching network (ITM), which leverages two auxiliary tasks, i.e., image-target relevance and object-target alignment, to improve the TMSC task. As shown in Fig. 3, ITM consists of four modules: Feature Extraction, Coarse-Grained Matching, Fine-Grained Matching, and Multimodal Fusion. We describe the details of each module in the following subsections.

4.1 Feature Extraction

Contextualized Target Representation. Given an input sentence S and its target T, we split S into two parts, i.e., the target and the remaining context, and combine them as the contextualized target input T′ . An example is shown in the bottom of Fig. 3.a, where we replace the target Emily in S with a special token $T$ as its context, and concatenate the context with Emily as the input. With the transformed target input T′ , we feed it to a widely-used pre-trained model RoBERTa [Liu et al., 2019] to obtain the contextualized target representation: HT = RoBERTa(T′ ), where HT ∈ Rd×n , d is the hidden dimension, and n is the input length.

(b) Alignment Distribution.

Image Representation. Given an image V, we use a widely-used object detection method Faster R-CNN [Ren et al., 2015] to detect object proposals and obtain their regional representations as our visual features [Anderson et al., 2018]. We then sort detected proposals by object category detection probabilities, and keep the top-100 object proposals in order to retain more small objects for target alignment. Let R = Faster R-CNN(V) denote the regional representations, where R ∈ R2048×100 . To model the interactions between objects, we feed R to Transformer to obtain object-level image representations: HV = Transformer(WTRR), where WR ∈ R2048×d and HV ∈ Rd×100 .

4.2 Coarse-GrainedMatching

The goal of this module is to capture the image-target relevance, and alleviate the noise from unrelated images.

To achieve this goal, we apply the Cross-Modal Transformer layer [Tsai et al., 2019] to model the interaction between the target and the image, which regards image representations HV as queries, and contextualized target representations HT as keys and values as follows:

where H′V ∈ Rd×100 is the generated target-based image representation. Next, we apply a max-pooling operator over H′V

to obtain the most salient features for relevance classification: h′V = max-pooling(H′V). Based on h′V, we use a Sigmoid function to perform image-target relevance classification:

We use the cross-entropy loss to optimize the image-target relevance task, denoted by Relevance (RE) Supervision:

where M is the number of samples in our annotated dataset.

Since the probability in Eqn. (2) is a scalar in the range of [0,1] indicating the relevant score between the target and the image, we use it to construct a visual filter matrix G ∈ Rd×100 , where each entry in G equals to P(r). With the visual filter matrix, we can obtain the filtered image representations as follows:

where ⊙ is the element-wise multiplication. For example, if G equals to 0, all the visual features are filtered.

4.3 Fine-GrainedMatching

Based on coarse-grained image-target matching, this FineGrained Matching module further aims to identify the finegrained visual objects aligned with the input target in those target-related images.

To achieve the object-target alignment, we apply another Cross-Modal Transformer layer to obtain the target-aware attention distribution over 100 object proposals from Faster RCNN. Specifically, we use the representation of the first token in the target input (i.e., H0T) as queries, and the filtered image representations H′′V as keys and values:

where H′T ∈ Rd×1 is the generated image-based target representations. Let us use Di to denote the attention weights in the i-th head attention of the Cross-Modal Transformer. We take the average of all the m-head attentions as the final distribution over 100 object proposals, denoted by D = 1m ∑mi=1 Di, where D ∈ R100 .

To guide the attention distribution to achieve object-target alignment, we propose to obtain an alignment distribution from ground-truth (GT) boxes as supervision. As shown in Fig 3.b, given an image, we first calculate the Intersection over Union (IoU) scores of its object proposals with respect to the GT bounding box, which denote the overlap between the proposal and GT bounding box. Following previous studies for visual grounding [Yu et al., 2018; Lei et al., 2020], for the i-th proposal, if its IoU score is larger than 0.5, we keep the IoU score and 0 otherwise. We can then get the IoU score distribution over all object proposals, denoted by [a1, ..., a100] ∈ R100 . Based on this, we renormalize the IoU score distribution to obtain the GT alignment distribution A ∈ R100 .

We adopt the Kullback-Leibler Divergence (KLD) loss to make the attention distribution D and the ground-truth alignment distribution A as close as possible, denoted by Attention (ATT) Supervision:

where C is the number of target-image related samples in our Image-Target Matching dataset.

4.4 Multimodal Fusion

With the image-based target representations H′T generated from the Fine-Grained Matching module, we concatenate it with the contextualized target representations as: HM = H′T ⊕ HT, and feed them to a Transformer layer for multimodal fusion:

Finally, the representation of the first token is fed to a softmax layer for sentiment classification:

The standard cross-entropy loss is to optimize the TMSC task, denoted by Sentiment Supervision:

where N is the number of samples for the TMSC task.

We employ the alternating optimization strategy to iteratively optimize the two auxiliary tasks with our Image-Target Matching dataset and optimize the main task with the dataset for TMSC. The combined objective function is:

where λ1 and λ2 are hyper-parameters.

5 Experiment

5.1 Experiment Setting

We adopt three datasets to systematically evaluate the effectiveness of our coarse-to-fine grained Image-Target Matching network (ITM). One is our Image-Target Matching dataset for the two auxiliary tasks, i.e., Image-Target Relevance and Object-Target Alignment, as introduced in Section 3. The other two are the benchmark Twitter datasets for the TMSC task, i.e., TWITTER-15 and TWITTER-17. The statistics of the two TMSC datasets are shown in Table 2.

For our ITM model, we adopt RoBERTabase [Liu et al., 2019] as the contextualized target encoder and Faster RCNN [Ren et al., 2015] with ResNet-101 backbone released by [Anderson et al., 2018] as the object detector. During the alternating optimization process, we use the AdamW optimizer, and fix the hyper-parameters after tuning them on the development set. Specifically, we set the batch size to 32, the training epoch to 10, and λ1 and λ2 to 1 and 0.5. The learning rates for the TMSC task and the two auxiliary tasks are set to 1e-5 and 1e-6 respectively.

5.2 Main Results

In this subsection, we compare our ITM model with several representative methods for TMSC, and report the accuracy (Acc) and the Macro-F1 score (F1) of each method in Table 3.

We first consider the following methods that focus on text only for comparison: 1) MGAN [Fan et al., 2018], a multigrained attention network capturing multi-level target-text interactions. 2) BERT [Devlin et al., 2019], a pre-trained model regarding target and text as a pair for sentiment classification. 3) RoBERTa [Liu et al., 2019], an enhanced pre-trained model based on BERT. Moreover, we consider the following multimodal approaches for comparison: 1) MIMN [Xu et al., 2019b], a multi-interactive memory network modeling the interaction between the target, text and image. 2) ESAFN [Yu et al., 2020], a target-sensitive attention and fusion network based on LSTM. 3) ViLBERT [Lu et al., 2019], a pre-trained Vision-Language model, in which the targettext pair is used as the textual input. 4) CapBERT [Khan et al., 2021], which translates the image to textual captions and combines the captions and the original target-text pair with a pre-trained BERT model. 5) TomBERT [Yu et al., 2019], a BERT-based TMSC approach with target-sensitive crossmodal attention. 6) CapRoBERTa, which replaces BERT with RoBERTa in CapBERT. 7) TomRoBERTa, which replaces BERT and ResNet with RoBERTa and Faster R-CNN in TomBERT. 8) TomRoBERTa+Aux-Tasks, a TomRoBERTabased multi-task learning baseline proposed by us, which adds our attention supervision in Eqn. (6) on their target attention layer and adds a softmax layer with relevance supervision over their final multimodal representation.

In Table 3, we can observe that RoBERTa achieves the best performance among text-only methods. It is reasonable since RoBERTa adopted better training strategies and larger corpus than BERT. For multimodal methods, it is easy to see that MIMN and ESAFN obtain the lowest performance, due to the lack of model pre-training. The pre-trained VL model (i.e., ViLBERT) performs worse than TomBERT, probably because the pre-trained dataset for ViLBERT is much smaller than BERT. Moreover, CapBERT performs better than all the other baseline systems, since it resorts to a pretrained image captioning model. It is intuitive that TomRoBERTa and CapRoBERTa generally performs better than TomBERT and CapBERT. In addition, it is surprising that TomRoBERTa+Aux-Tasks performs even worse than TomRoBERTa. We conjecture the reason is: 1) its target attention layer only uses the target without its context as the target input; 2) due to the structure of TomBERT, the object-target alignment is performed before the image-target relevance, which may bring much visual noise to object-target alignment. Finally, we can clearly see that ITM achieves the best results on both accuracy and F1 score among all the compared systems across the two datasets. These observations demonstrate the effectiveness of our ITMmodel and the importance of incorporating image-target matching for the TMSC task.

5.3 Results of Image-Target Matching

Table 4 shows the results of Image-Target Relevance Classification on our Image-Target Matching dataset in Section 3. It can be seen that our ITM model significantly outperforms TomRoBERTa+Aux-Tasks on all the metrics, showing the advantage of ITM for Image-Target Relevance.

Table 5 shows the results of Object-Target Alignment on our Image-Target Matching dataset. The evaluation metrics are Kullback-Leibler Divergence (KLD) between the attention distribution D and the ground-truth alignment distribution A in Section 4.3 and the recall of the top-ranked bounding box in A from top-k bounding boxes in D, denoted by R@k. In Table 5, it is clear that ITM significantly outperforms TomRoBERTa+Aux-Tasks in terms of all the metrics, showing the advantage of ITM for Object-Target Alignment.

5.4 In-depth Analysis

Ablation Study. We explore the impact of different components in our model and report the results in Table 6. Specifically, removing either the relevance supervision in Eqn. (3) or the attention supervision in Eqn. (6) leads to a moderate performance drop on both accuracy and F1 score. Moreover, discarding the two supervisions will lead to a significant performance drop of around 1.6 percentage points on accuracy and 3 percentage points on F1 score. These observations indicate the indispensable effects of filtering the visual noise and achieving object-target alignment. Lastly, from the last three rows of Table 6, we find that removing either coarse or finegrained matching module or both modules in Section 4.2 and Section 4.3 consistently decreases the performance, which indicates the necessity of incorporating Cross-Modal Transformer layers to achieve cross-modal alignments.

Case Study. In the left two columns of Table 7, we show two representative test samples to demonstrate the importance of filtering the unrelated images. For case (a), given the target Pacific Rim, RoBERTa accurately predicted its sentiment as Positive, while TomRoBERTa gave the wrong prediction after combining the unrelated image. In contrast, ITM gave the correct sentiment prediction and a low image-target relevance score as well as an evenly distributed alignment distribution (i.e. no object is obviously aligned with Pacific Rim in the image). Similarly, for case (b), given Stagecoach as the target, TomRoBERTa wrongly predicted its sentiment as Positive due to the unrelated image, while ITM correctly predicted the Neutral sentiment after filtering the visual noise.

Effects of Image-Target Relevance

Effects of Object-Target Alignment

In the right two columns of Table 7, we use another two test samples to show the effect of achieving object-target alignment. For case (c), we can see that it is in the same tweet in case (b), but the given target is changed to SamHunt. In this case, RoBERTa and TomRoBERTa still made the same predictions as before, while ITM generated the appropriate relevance score and assigned higher attention weights to the objects around SamHunt, and thus gave the correct sentiment. For case (d), given the target Joe Flacco, RoBERTa wrongly predicted its sentiment as Neutral due to the absence of sentiment words. TomRoBERTa also gave the wrong prediction, because it failed to identify the small object about Joe Flacco. By contrast, our ITMmodel correctly predicted the sentiment as Positive with more attention on his smiling face.

6 RelatedWork

Targeted sentiment classification (TSC, a.k.a aspect-based sentiment classification) has been well studied in recent years. Various traditional feature-based models [Jiang et al., 2011; Pontiki et al., 2016] and deep learning-based models [Tang et al., 2016; Wang et al., 2018; Xu et al., 2020] have been proposed to address the TSC task. More recently, many Transformer-based methods [Xu et al., 2019a; Dai et al., 2021] and graph neural network-based methods [Wang et al., 2020; Tang et al., 2020] are designed to better leverage sequential and syntactic information for the task. Despite obtaining remarkable results, these approaches failed to consider the information from other modalities, e.g., images.

With the explosive growth of multimodal data, multimodal sentiment analysis (MSA) has attracted wide attention recently. For coarse-grained MSA, many approaches have explored the capability of adopting neural networks to build the interactions between modalities for MSA in conversations or tweets [Poria et al., 2017; Xu et al., 2018;

Zhang et al., 2020; Yang et al., 2021]. For fine-grained MSA, various LSTM-based and Transformer-based methods were proposed to capture the fine-grained interaction across different modalities for the TMSC task [Xu et al., 2019b; Yu et al., 2020]. In this work, we follow the later line of work, aiming to improve TMSC with Image-Target Matching.

Since our Object-Target Alignment task is closely related to Visual Grounding (VG), we review some representative studies for VG, which aim to predict the location of an image region referred by the language expression. Earlier works to VG primarily focus on selecting visual objects based on parsing linguistic descriptions [Kazemzadeh et al., 2014; Yu et al., 2016]. Recently, a myriad of visual-language pre-training models have been proposed for VG to capture the alignment between image and language modalities [Su et al., 2019; Yu et al., 2021].

7 Conclusion

In this paper, we proposed a multi-task learning model named coarse-to-fine grained Image-Target Matching network (ITM), which leverages two auxiliary tasks, i.e., ImageTarget Relevance and Object-Target Alignment, to capture the image-target matching relations for the TMSC task. Experiment results on two TMSC datasets and our Image-Target Matching dataset demonstrate that our ITM model consistently outperforms a number of state-of-the-art methods.

Acknowledgments

This work was supported by the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018), and the Natural Science Foundation of China (62076133 and 62006117).