SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities

1 Introduction

The present deep learning algorithms thrive on a wide spectrum of real-life tasks, such as those in the fields of computer vision and natural language processing, for which the essence is to learn the underlying distribution from the dataset. Inspired by these achievements, researchers march on to conquer the more complex tasks that put emphasis on reasoning and knowledge. One character of these complex tasks is that they normally involve data from distinct modalities.

∗Co-first authors.

†The corresponding author.

For example, in the Visual Question Answering (VQA) task, one must answer the questions based on the knowledge extracted from the corresponding image [Goyal et al., 2017; Antol et al., 2015]. The ability to fuse multi-modal information serves as the cornerstone to developing solutions to those complex tasks, and it presents severe challenges to the deep learning community.

Recently, many large multi-modal datasets have been collected, e.g. [Vadicamo et al., 2017; Zadeh et al., 2018; Bowman et al., 2015; Liang et al., 2021], laying the ground for exploring sophisticated fusion strategies. Beyond the naive methods of early and late fusion strategies, increasing attention has been paid to developing and practicing hybrid fusing strategies [Wang, 2021; Baltruˇsaitis et al., 2018; Wang et al., 2020; Bai et al., 2021]. Those efforts constantly push forward the model performance on the multimodal datasets. Despite the success, people gradually realize a subtle limitation of the present progresses: the model performance is almost solely judged on their prediction accuracy. A higher accuracy does not necessarily translate to the better usage of multi-modal information. For example, in the CMU-MOSEI datasets, the accuracy of a unimodal model trained on the text modality is very close to the ones of SOTA multi-modal models [Zadeh et al., 2018; Delbrouck et al., 2020]. Consequently, there is an urgent need for a quantitative metric that can capture how the underlying models operate on the multi-modal information.

To evaluate the individual modalities' importance, Gat et al. [2021] proposed the perceptual score, which estimates the contribution of a modality with its marginal performance gain at the presence of other modalities. However, this score cannot measure cross-modal cooperation and, in the presence of more than two modalities, its attribution of the importance to individual modalities does not correctly reflect their marginal contribution. To address the above issues, we draw inspirations from the game theory and propose the SHapley vAlue-based PErceptual (SHAPE) scores to directly evaluate the marginal contribution and the cooperation of individual modalities. The validity of the SHAPE scores roots in the properties of the celebrated Shapley value [Shapley, 1953].

1 The full version of our paper: https://github.com/zhouyilab/ SHAPE/blob/main/SHAPE-IJCAI22.pdf; Implementation: https:// github.com/zhouyilab/SHAPE.

With the SHAPE scores, we aim at providing additional perspectives for evaluating the performance of the multi-modal models, which in turn would help building sophisticated fusion strategies.

Our contributions are three-fold:

1. We propose the SHAPE scores, which are theoretically guaranteed to quantify the marginal contribution of individual modalities and the cross-modal cooperation.
2. With the SHAPE scores, we systematically evaluate the usage of multi-modal information by models using different fusion methods on the different multi-modal datasets for different tasks.
3. We extend the perceptual score to a more informative fine-grained version. Moreover, we demonstrate that the perceptual score implicitly measures the correlation among modalities within the dataset.

2 RelatedWork

2.1 Multi-modal FusionMethod

The fusion strategies can be roughly classified as early fusion, late fusion, and hybrid fusion [Baltruˇsaitis et al., 2018]. In the early fusion, one directly concatenates inputs from individual modalities to build a unified input embedding. In this case, a single pathway model is enough to make a prediction, making the training pipeline easier [Kiela et al., 2019]. The late fusion usually processes individual modalities in different pathways. The fusion operation is typically implemented on the output from each pathway. The later fusion strategy ignores the low-level feature interaction, making it hard to learn the joint representation of modalities. However, in practice, later fusion enables a model using the different pre-trained models for different pathways, making it more flexible than the early fusion [Rodrigues Makiuchi et al., 2019]. Hybrid fusion has no specific recipe. The different modalities can be integrated at different stages, on different feature levels, and among different modules. It could effectively promote collaboration among individual modalities. Generally, the optimal hybrid fusion strategy depends on the type of the underlying task. For example, in the VQA task, the co-attention-based fusion methods are proved to be better than other strategies [Liu et al., 2019].

2.2 Modality Contribution Attribution

Presently, most evaluation works focus on the feature importance. For example, Lundberg and Lee [2017] proposed the SHapley Additive exPlanations (SHAP) value, which can estimate feature importance using an approximation of the features Shapley values. For evaluating modal importance, Hessel and Lee [2020] adopted an indirect attribution method. They designed a tool called the empirical multimodallyadditive function projection (EMAP), which is an effective model that elimates the cross-modal cooperation in the reference multi-modal model. Through comparing the performance of the reference model and its EMAP, one could infer how much the cross-modal cooperation contribute.

The work most relevant to ours is the perceptual score, which is proposed by Gat et al. [2021] to assess the importance of a modality in the input data for a given model.

They adopted a permutation-based approach for removing the modality influence. Specifically, let xM denote the feature of target modality M in the input data point. In order to remove M's influence, xM is substituted with a randomly chosen one from another data point, i.e., x′M . The resulting classifier on the permutated data is called the permuted classifier. The perceptual score for M is defined as the normalized difference between the accuracies of the normal classifier and the corresponding permutated classifier.

It is worth noting that the perceptual score does not account for the marginal contribution of the target modality M. Instead, it measures the M's contribution conditioned on all other modalities in the dataset. The two types of contributions are the same only in the two-modal case. In the presence of more modalities, it requires a more sophisticated method to compute the marginal contribution. Moreover, there still lacks a quantitative method that isolates the cooperation between different modalities.

2.3 Shapley Value

The Shapley value [Shapley, 1953] was initially proposed in the coalitional game theory for fair attribution of payouts among the players depending on their contribution to the total payout. A multi-player game is usually defined as a real-valued utility function V on its participants. Let N := {1, . . . , n} denote the set of players in the game. The player can form different coalitions (i.e., subsets of N ) whose net effect can be either cooperative or competing. Consider a player i and a subset S ⊆ N/{i}, the marginal contribution of i to S can be evaluated by

Since a player can interact with any other players, in order to evaluate the marginal contribution of i in the game V (N) one needs to average over all possible mi(S; V ). This is exactly how the Shapley value φi is defined,

A fair attribution of payouts is considered to satisfy the properties of Efficiency, Symmetry, Dummy and Additivity, and the Shapley value is the only attribution method that has those four properites [Molnar, 2020]. More precisely,

Efficiency. All the individual rewards sum to the overall net payouts, i.e., ∑i φi = V (N) - V (∅), where V (∅) indicates the baseline reward without any player.

Symmetry. If two palyers i, j contribute equally to all possible coalitions, i.e., V (S ∪ {i}) = V (S ∪ {j}), ∀S ⊆ N/{i, j}, then they have the same reward, φi = φj.

Dummy. A dummy player i, which does not contribute to any coalition, i.e., V (S ∪ {i}) = V (S) + V ({i}), ∀S ⊆ N/{i}, has the reward as if playing alone, φi = V ({i}).

Additivity. Consider a combined game U(S) = V (S) + W(S), ∀S ⊆ N, its reward is the sum of the rewards for individual games, i.e. φi(U) = φi(V ) + φi(W).

Those four properties in turn could be viewed as axioms defining a fair attribution method, and it has been proved that they uniquely lead to the definition of the Shapley value [Weber, 1988].

3 Method

3.1 Models for Comparison

In deep learning, diverse fusion methods have been developed to address the multi-modal learning tasks [Baltruˇsaitis et al., 2018; Wang, 2021]. Among them, the Early Fusion and Late Fusion strategies are widely recognized. Besides these two, the rest methods cannot be easily categorized. Depending on specific implementation, the fusion may happen at one particular stage or at multiple stages. In addition, different works usually adopt distinct network architectures, making it infeasible to compare them on the same ground.

The present work is devoted to evaluate the contribution and cooperation of individual modalities under different fusion methods on different multi-modal datasets for different tasks. To achieve this goal, we need (i) a common network architecture that ensures decent performance on different datasets for different tasks. (ii) a unified framework to account for different fusion types.

To address point (i), we adopt the transformer [Vaswani et al., 2017] architecture, which is capable of handling different modalities. We are aware that using a common architecture for different tasks would potentially degrade the performance. For point (ii), we introduce the Dense Fusion strategy that builds upon the transformer architecture (see Figure 1). The fusion happens at every stage. We also consider one of its variations, which includes weights (λ's) to control the fusion intensity. Those weights are learned during the training. Such a variation is referred as the Dynamic Dense Fusion.

To summarize, in our experiments, we consider the Early Fusion, Late Fusion, as well as two variations of the Dense Fusion strategy. All the network models are built upon the transformer architecture. Details can be found in the experiment setup and in the supplementary material.

3.2 The SHAPE Scores

The SHAPE scores are built upon the Shapley value with identifying the set of modalities as players and model accuracy as the utility function. We distinguish two types of SHAPE scores, one captures the marginal contribution of individual modalities, and the other captures the cross-modal cooperation. The SHAPE scores are easy to compute under the general multi-modal learning scenario, no matter how many modalities are involved.

In the following, we use Vf(D) to denote the accuracy of model f on the dataset D. When discussing the contribution of different modalities, e.g., M1 and M2, we shall slightly change the symbol to Vf(M1,M2) for emphasizing the involved modalities. The set of modalities in D is denoted as M := {M1, . . . ,Mm}.

SHAPE Score for Modality's Marginal Contribution

For the purpose of evaluating the marginal contribution of a modality Mi, we use the scaled Shapley value

where the normalization factor Zf is introduced to align the scores for different models and thus enable cross-model comparison. Similar to the modal-normalization in [Gat et al., 2021], we choose Zf to be the model accuracy on the full dataset D, i.e., Zf = Vf(D). For simplicity, we will use the abbreviated symbol SMi , when there is no risk of confusion.

As an example, for the two-modal case, we have

where Zf = V (M1,M2). Extensions to three or more modalities cases are straightforward, and we leave them to the supplementary material.

Baseline Values. The above 0i refers to the absence of modality Mi and should be implemented using some predefined baseline values. In the case where only one modality is absent, it is natural to substitute zeros for the missing inputs. However, when both modalities are absent, i.e., V (01, 02), an all-zero input would produce a single deterministic prediction that is sensitive to specific training settings, such as the model initialization. We argue that it is better to predict the majority class in this case. This majority-classifier is stable and could account for the possible unbalances in the dataset.

SHAPE Score for Cross-modal Cooperation

The Shapley value also enables us to directly measure the cross-modal cooperation, which evaluates the degree to which the model exploits the complementary information of a modality provided to the other modalities.

Intuitively, to compute the cooperation among a set of modalities A ⊆ M, we need to subtract the marginal contributions of all Mi ∈ A from the marginal contribution of A itself. Treating A as a single effective player, its marginal contribution can be evaluated by φA(M; Vf). On the other hand, the marginal contribution of Mi ∈ A should be φMi (M/A ∪ Mi; Vf). Consequently, the cooperation score is defined as

Note that the set of "players" in the above φMi is different from the one in Equation (1). For simplicity, we usually omit the subscripts f ; D and write the score as CA.

For a concrete example, let's consider the two-modal case. After a few steps of simple algebra, we arrive at

where 01 and 02 denote the baseline values for modalities M1,M2, respectively.

3.3 Fine-grained Perceptual Scores

In this subsection, we proceed to have a closer look at the perceptual score [Gat et al., 2021]. First, we will show that this score implicitly measures the correlation among different modalities within the dataset D. Then we propose a finegrained version of the perceptual score. In the following, we only keep the key steps and leave the detailed deduction to the supplementary material.

Let (x, y) denote a data sample in D, where x ∈ X is the input feature and y ∈ {0, . . . , k - 1} is the corresponding label. In order to explicitly refer the modalities, we will decompose x as (xM1 , xM2 ), where M1 ⊆ M and M2 = M/M1 are the subsets of modalities. Recall that the model accuracy is defined as Vf(D) = ED[Iy∗=y], where I is the identity function and y∗ refers to the model prediction at input x. Simple algebra will lead us to

where Pr(y∗, xM1 , xM2 ) is the probability distribution on D. The permutation-based step in the perceptual score would alter the data distribution, resulting in a new effective dataset with distribution Pr(y, xM1 )Pr(xM2 ). We refer to this effective dataset as Dp . Now, the perceptual score forM2 is

where ∆M1,M2;f = P(y∗ , xM1 |xM2 ) - P(y∗ , xM1 ). Clearly, apart from the dependence on the model f through y∗ , the term ∆M1,M2;f effectively measure the correlation betweenM1 andM2 in the dataset D.

Since P(xM2 ) is non-negative, if the part xM2 comes from the same class as xM1 , then we would expect ∆M1,M2;f ≥ 0. We refer to this case as the in-class permutation. Likewise, in the out-class permutation, xM2 comes from a different class from xM1 and we expect ∆M1,M2;f ≤ 0.

The in-class and out-class cases provide conflicting driving forces that would cancel each other and cannot be reflected by the original perceptual score. Based on this observation, we propose to use the in-class and out-class perceptual scores in place of the original one. They are defined as following:

where the [M/Mi]± operator removes theMi modality either by the in-class permutation (+) or the out-class permutation (-). Zf is the normalization factor as usual. We only consider the model-normalization here. Finally, for simplicity, we usually omit the subscripts f; D as long as it raises no confusion.

4 Experiments

4.1 Setup

For all models, we use 4-layer transformer encoder as the backbone. The number of attention heads is 8, and the hidden dimension is 1024. All models are trained using the Adam optimizer with learning rate warm-up in the first epoch. We encode text data with the GloVe [Pennington et al., 2014], and audio data with the MFCC [Tiwari, 2010]. As for the image data, we employ the patch embedding technique, following the ViT Model [Dosovitskiy et al., 2020]. It is worth noting that our models achieve comparable performance with the supervision-based SOTA models.

4.2 Datasets

Real-life applications raise diverse demands for multi-modal data. Depending on specific tasks, some must be handled by combining all the involved modalities, while in the others, additional modalities merely provide complementary information. Furthermore, different modalities in a multi-modal dataset might be only weakly correlated to each other, or they could be different representations that provide precisely the same information. To account for those complicated situations, we deliberately choose three datasets, as will be introduced below.

B-T4SA. The dataset [Vadicamo et al., 2017] is collected from Twitter, including the posted text (textual modality T) and associated images (visual modality V). Each data pair is annotated with a sentiment label (positive, negative and neutral). The task is to predict this label using both the text and image inputs. We note that the text data normally align with their sentiment labels while a large number of images seem to be irrelevant to the labels. B-T4SA represents the situation that different modalities are complementary and weakly correlated to each other.

CMU-MOSEI. This dataset [Zadeh et al., 2018] is the largest dataset for sentence-level sentiment analysis and emotion recognition. It contains more than 65 hours of annotated video data, consisting of audio, video and text modalities. In our experiments, we only use the audio (A) and text (T) modalities. The text modality includes the scripts that are generated from the audio data. In this way, the CMU-MOSEI dataset stands for the situation that different modalities are complementary and highly correlated.

SNLI. The SNLI dataset [Bowman et al., 2015] is a collection of 570k human-written English sentence pairs. Each pair contains a premise and a hypothesis, whose relation is labeled as either entailment, contradiction, or neutral. We shall view the premise and hypothesis as two distinct modalities, i.e., Tp and Th. The underlying task is to infer the relationship between the premise and the hypothesis. Clearly, this task intrinsically relies on both modalities and, thus, represents the situation that different modalities are indispensable.

4.3 Contribution and Cooperation

In this subsection, we investigate the SHAPE scores across different models on the three multi-modal datasets, demonstrating their ability in capturing the marginal contribution of individual modalities and the cross-modal cooperation.

Table 1 summarizes the results for B-T4SA dataset. We find the unimodal model for T performs as good as multimodal models, while the unimodal model for V only slightly suppasses the random guess. The SHAPE scores align with these observations. The contribution scores SV for V modality are near zero for all the fusion strategies, indicating V almost does not contribute to the task. Further, the cooperation scores CV T are very small as well. Hence, as expected, nearly no cooperation happens between the two modalities. At last, we note that CV T for early fusion and dense fusion are evidently higher than the other two models. This may be attributed to the fact that both the early and the dense fusion models encourage the fusion between the two modalities.

The A modality in the CMU-MOSEI dataset encodes richer information than its text counterparts by including features such as the tone and the talking speed. On the other hand, the T modality is better at representing semantic information. One would expect a multi-modal model to actively integrate the two modalities and make better predictions. Unfortunately, it is not the case as indicated by the SHAPE scores (see Table 2). Values of SA and CAT are very small, indicating the audio data seldom contribute to the model performance and there is little cooperation between the A and T modalities. In addition, the marginal contributions of T modality, ST , are also relatively small. At first sight, it may seem contradictory that both modalities have limited contribution to the performance, while the model achieves a decent accuracy. Closer investigation reveals that this is a consequence of the unbalanced dataset. Simply by predicting positive, one could get an accuracy of 71.04%, which is the accuracy of the unimodal model for A. This phenomenon coincides with the common belief that the deep neural networks are lazy, tending to take shortcuts. It could be the case that the multi-modal models simply exploit the unbalance of the dataset and pay little effort in mining the cooperation.

In Table 3, we show the results for the natural language inference task on the SNLI dataset, for which both modalities are indispensable. We find that both unimodal models perform poorly. Interestingly, the unimodal model for Th significantly outperforms the random guess. This implies that there could be implicit biases within the dataset despite that it is balanced. As expected, the behavior of SHAPE scores is different from previous cases. STp and STh becomes comparable, with STh being generally higher. Hence, both modalities contribute, and the hypothesis is more important for model prediction. More strikingly, the cooperation scores CTpTh are also high. This suggests that the models actively exploit the cooperation between Tp and Th for predictions. At last, in the late fusion case, STp and CTpTh are much lower than the other cases. The lack of cooperation could be attributed to the nature of the late fusion strategy, i.e., the fusion only happen at the last step. This helps explain the low accuracy of the late fusion model and suggest that, in this case, fusion at earlier stages is a better choice.

4.4 Compared with the Perceptual Score

In this subsection, we compare our SHAPE scores with the perceptual score as well as with its fine-grained versions.

Firstly, we observe that the perceptual score P coincides with our marginal contribution SHAPE score S in the sense that their values are highly correlated. In the B-T4SA dataset, both SV and PV are close to zero, and ST (PT ) is significantly higher than SV (PV ). Similar pattern is found in the CMU-MOSEI dataset. For the SNLI dataset, S and P are both considerably higher than zero, except for the Late Fusion case. Besides, both scores for Th modality are higher than the ones for Tp modality.

Despite the similarities, we emphasize the superiority of our SHAPE scores for (i) guaranteed by the game theory, the SHAPE score S can correctly capture the marginal contribution no matter how many modalities are involved. (ii) the SHAPE score C can directly quantify the cooperation among modalities in addition to their individual importance.

Next, we turn to the fine-grained perceptual scores. As argued in section 3.3, the raw perceptual score implicitly measures the correlation among modalities in the dataset. However, the positive and negative correlations are mixed and cancel each other, causing a considerable loss of information. To be concrete, let's consider the Late Fusion case for the SNLI dataset. With only the raw perceptual scores, we at best might conclude that the Th modality is more important than the Tp one in the contribution to model performance. On the other hand, we find P in Th is much smaller than P out Th , indicating that the model prediction almost relies solely on the Th modality. The small values of P in Tp and P out Tp further consolidate this conclusion. Putting it differently, the Late Fusion model tends to use only the Th modality while ignoring the Tp modality. Consequently, the cooperation between the two modalities should be small in the Late Fusion case. This is just what the cooperation SHAPE score CTpTh reveals to us. Under more complicated scenario with three or more modalities, it would be harder to capture the cross-modal cooperation using perceptual scores. On the contrary, it is always straightforward to quantify the cross-modal cooperation with the SHAPE score C.

5 Discussion

The above experiment results suggest that the way a model exploits multi-modal information depends on various factors, including the intrinsic correlation among individual modalities in the dataset, the type of the task, and the fusion strategy in use. Based on this observation, we want to draw attention to dataset and task in multi-modal learning research. There is little hope to train a model that can exploit crossmodal cooperation on a dataset where different modalities are nearly independent. Likewise, if the task can be easily handled with only a single modality, one must try hard to design an algorithm to exploit the possible cross-modal cooperation. Furthermore, the bias and unbalance in dataset also affect multi-modal learning. In many cases, the improvement in the accuracy of a multi-modal model could be illusive. Our SHAPE score C can directly quantify the cooperation among individual modalities and, hence, could help diagnose the fusion strategy.

At last, it is worth noting that though the present SHAPE scores are defined using the accuracy metric as the utility function, there is no limitation of what function should be used. In fact, the SHAPE scores can readily adapt to more complex metrics such as the F1 score.

6 Conclusion

In this paper, we present the SHAPE scores that quantify the marginal contribution of individual modalities and their collaboration. Properties of the Shapely value provides theoretical guarantee for the validity of our SHAPE scores and the experiments empirically verify their effectiveness. We would like to suggest using the SHAPE scores in addition to the accuracy metric when analyzing a multi-modal model, as they are easy to compute and can extract key information about how the model use multi-modal data.

We are aware that the present SHAPE scores are not yet perfect. The choice of baseline values deserve more deliberate consideration. In the future work, we will keep improving the SHAPE scores and systematically evaluate the SOTA multi-modal models.

Acknowledgments

This work was supported by grants from the National Science and Technology Innovation 2030 Project of China to Yi Zhou (2021ZD0202600).