CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation

1 Introduction

Making abstraction from an image into high-level semantics is one of the most remarkable capabilities of humans. Scene Graph Generation (SGG) [Xu et al., 2017] - a task of extracting objects and their relationships in an image to form a graphical representation - aims to achieve the abstraction capability and bridge the gap between vision and language. SGG has greatly benefited the down-stream tasks of question answering [Norcliffe-Brown et al., 2018;

∗Equal contribution.

†Corresponding author.

Figure 1: (a) SGGmodel with conventional flat loss. (b) SGGmodel with our proposed cognition tree loss. The word size in the top-left box is proportional to the relationship frequency.

Zhu et al., 2020] and visual understanding [Shi et al., 2019; Jiang et al., 2020]. Some works [Zhu et al., 2020; Jiang et al., 2020] feed the scene graphs into graph neural networks for relation-aware object representation. Some others [Hudson and Manning, 2019] perform sequential reasoning by traversing the relational graphs. Compared with independent objects, the rich relationships benefit explainable reasoning.

Although much effort has been made with high accuracy in object detection, the detected relationships are far from satisfaction due to the long-tailed data distribution. Only a small portion of the relationships have abundant samples (head) while most ones contain just a few (tail). This heavily biased training data causes biased relationship prediction. Tail relationships will be mostly predicted into head classes, which are not discriminative enough for high-level reasoning, such as falsely predicting on instead of looking at, and coarsely predicting on instead of walking on. Besides, tail relationships like standing on, sitting on and lying on can be even harder to distinguish from each other due to their visual similarity and scarce training data.

To tackle this problem, most research focuses on learning unbiased models by re-weighting losses [Zareian et al., 2020] or disentangling unbiased representations [Tang et al., 2020]. However, humans can effectively infer the correct relationships even when some relationships appear more frequently than others. The essential difference between human and AI systems that has been ignored lies neither in the learning strategy nor feature representation, but in the way that the concepts are organized. To illustrate this discrepancy, we show an example truck parked on street in Figure 1. Existing models treat all the relationships independently for flat classification (Figure 1(a)). Because of the data bias, the head relationships, e.g. on and near, obtain high predicted probability. In contrast, the cognition theory [Sarafyazd and Jazayeri, 2019] supports that humans process information hierarchically in the prefrontal cortex (PFC). We intuitively start from making a rough distinction between remarkably different relationships. As shown in Figure 1(b), relationships belonging to the concept "on", e.g. on and parked on, will firstly be distinguished from the ones about "near", e.g. near and behind; then we move to distinguish the slight discrepancy among easily confused ones in one concept, e.g. parked on, standing on and walking on.

Inspired by the hierarchical reasoning mechanism in PFC, we propose a novel loss function, Cognition Tree (CogTree) loss, for unbiased scene graph generation. We first propose to build a hierarchy of the relationships, imitating the knowledge structure built from the independent relationships in human mind. The CogTree is derived from the prediction of a biased SGG model that satisfies the aforementioned thinking principles: distinguishing remarkably different relationships at first and then focusing on a small portion of easily confused ones. Then we design a CogTree-based loss to train the SGG network from scratch. This loss enables the network to surpass the noises from inter-concept relationships and then intra-concept relationships progressively. It frees the SGG models from the burden of distinguishing detailed discrepancy among all the relationships at one time, resulting in more accurate prediction via this coarse-to-fine strategy.

The main contributions are summarized as follows: (1) We propose a debiasing loss based on the inductive hierarchical structure, which supports coarse-to-fine distinction for the correct relationships while progressively eliminating the interference of irrelevant ones. (2) We automatically build the cognitive structure of the relationships from the biased SGG predictions, which reveals the inherent hierarchy that the relationships are organized after biased training; (3) The proposed loss is model-agnostic and can be applied to several representative SGG models. The results of extensive experiments indicate that the CogTree loss remarkably alleviates the bias in SGG and our loss achieves a new state-of-the-art performance compared with existing debiasing methods.

2 RelatedWork

Scene Graph Generation. SGG [Xu et al., 2017] products graphical abstraction of an image and encourages visual relational reasoning and understanding in various down-stream tasks [Shi et al., 2019; Zhu et al., 2020]. Early works focus on object detection and relationship detection via independent networks [Lu et al., 2016; Zhang et al., 2017], ignoring the rich contextual information. To incorporate the global visual context, recent works leverage message passing mechanism [Xu et al., 2017; Chen et al., 2019a], recurrent sequential architectures [Zellers et al., 2018; Tang et al., 2019] and contrastive learning [Zhang et al., 2019] for more discriminative object and relationship representations. Although the accuracy is high in object detection, the relationship detection is far from satisfaction due to the heavily biased data. [Chen et al., 2019b; Tang et al., 2019] consider the biased SGG problem and propose mean Recall as the unbiased metric without corresponding debiasing solutions. The recent work [Liang et al., 2019] prunes the predominant relationships and keeps the tail but informative ones in the dataset. [Tang et al., 2020] proposes the first solution for unbiased SGG by counterfactual surgeries on causal graphs. We rethink SGG task from the cognition view and novelly solve the debiasing problem based on the coarse-to-fine structure of the relationships.

Biased Classification. Classification on highly-biased training data has been extensively studied in previous work, which can be divided into three categories: (1) balancing data distribution by data augmentation or re-sampling [Burnaev et al., 2015; Li and Vasconcelos, 2019]; (2) debiasing learning strategies by re-weighting losses or training curriculums [Lin et al., 2017b; Cui et al., 2019]; (3) separating biased representations from the unbiased for prediction [Cadene et al., 2019; Tang et al., 2020]. Our CogTree loss belongs to the second category but differs from existing methods in that we are the first to leverage the hierarchical structure inherent in the relationships for re-weighting, which enables more discriminative representation learning by a coarse-to-fine mode.

3 Methodology

We first introduce the automatic CogTree building process. Then a novel debiasing loss based on the tree structure is proposed to train the SGG models. Since the CogTree loss is model-agnostic and applicable to various SGG models, we case-study two representative models, i.e. the widely compared MOTIFS [Zellers et al., 2018] and the state-of-the-art VCTree [Tang et al., 2019], and a newly proposed model, which first applies the transformer [Vaswani et al., 2017] architecture to the SGG scenario and achieves superior performance. We name it as SG-Transformer and take it as a strong baseline. The framework is illustrated in Figure 2.

3.1 Automatic CogTree Building

Once the SGG model has been trained on the long-tailed data, the model tends to be biased on predicting "head" relationships. When different samples are predicted with the same relationship, they mostly share similar properties on either visual appearance (e.g. walking on and standing on) or high-level semantics (e.g. has and with). We regard these relationships sharing common properties as in one concept. We automatically induce concepts from the biased predictions and re-organize the relationships based on the concepts by a tree structure, denoted as Cognition Tree (CogTree). As shown on the top of Figure 2, this automatic process contains the following three steps:

Step 1: Bias-Based Concept Induction. For all the samples in the ground-truth class di, we predict their labels via a biased model and calculate the distribution of predicted label frequency, denoted as Pi. The most frequently predicted class dj is regarded as the concept relationship of the ground-truth class di. As shown in Figure 2(Step 1), on is the concept relationship of itself, standing on and walking on. The above operation induces all the relationships into C concepts with corresponding concept relationships {ci}C .

Step 2: Concept-Centered Subtree Building. We reorganize all the relationships in each concept by a ConceptCentered Subtree. For the ith subtree, the root is ci while the leaves are the fine-grained relationships induced in ci. Note that, if a subtree only contains a root ci, e.g. parked on in Figure 2 (Step 2(A)), then ci is considered not representative enough to summarize common properties. We further induce it to the most approximate concept relationship, which has the second highest frequency in Pi, e.g. induce parked on to on in Figure 2 (Step 2(B)). This process outputs T subtrees.

Step 3: Cognition-Based Subtree Aggregation. We aggregate the T subtrees into one hierarchical CogTree. As illustrated in Figure 3, CogTree contains four functional layers and each layer induces relationships into coarser groups than the layer that comes after it. Specifically, the Root Layer (y0) contains a virtual node. The Concept Layer (y1) distinguishes which concept the input belongs to. It contains T virtual nodes, each representing a subtree induced in Step 2. The following Coarse-fine Layer (y2) tells whether the input can be described by a coarse-grained or fine-grained relationship in a certain concept. To this end, y2 splits each node in y1 into two nodes: one leaf node indicating the concept relationship while the other virtual node representing the cluster of fine-grained relationships in that concept. Each virtual node in y2 links to its fine-grained relationships of the corresponding concept in the Fine-grained Layer (y3). y3 focuses on distinguishing the slight discrepancy among easily confused relationships, e.g. standing on and walking on.

3.2 Debiasing CogTree Loss

We propose a CogTree loss for training the SGG model for unbiased predictions. This loss encourages the model to classify relationships from coarse to fine according to the CogTree structure (see the bottom right of Figure 2). To better make the loss functions (3) and (4) understood, several notations are provided in advance as below:

Ground-Truth CogTree Path (GCP). It denotes the ground-truth classification path in CogTree for a training sample. Given the ground-truth label of a sample, we track the path from the root to the leaf with that label and denote this path as the ground-truth path Lpath = {l0, l1, ...lM}, where the kth value lm is the ground-truth node at layer ym.

Predicted CogTree Probability (PCP). It means the predicted probability of each node in CogTree for a given sample by a biased SGG model. We denote the predicted probability set of all the classes as Ppred and denote the probability of class k as pk. The PCP of node i is defined as below:

where C(i) means the children of node i. class(i) denotes the corresponding class of node i.

Class-Balanced Weight (CBW). It denotes the balance weight of each node in CogTree. Since the success of reweighting strategy for debiasing, we adopt a well performed weighting factor [Cui et al., 2019] to compute CBW of each leaf and uniformly define CBW of each node as below:

where β is a hyper-parameter in the range of [0, 1) and ni is the sample number of the corresponding class of leaf i.

CogTree Loss. It contains two parts for complementary benefits: CogTree-based class-balanced (TCB) loss for debiasing by coarse-to-fine relationship classification and classbalanced (CB) loss for debiasing by re-weighting on softmax cross-entropy loss. The two loss terms are defined below.

Given a sample with the ground-truth path Lpath, as illustrated in Figure 3, we compute the class-balanced softmax cross-entropy loss on each layer of CogTree, and average the results to obtain the TCB loss as below:

where B(i) means the brothers of node i. TCB loss forces the network to surpass the noises from inter-concept relationships and learn concept-specific embeddings first, and then surpass the noises within one concept to refine relationship-specific embeddings, resulting in more fine-grained predictions.

Given a sample with the ground-truth class k, CB loss computes the class-balanced softmax cross-entropy based on the biased predicted probabilities Ppred:

where wk is the weighting factor [Cui et al., 2019] for class k. Empirically, we found it the best to combine TCB with CB. In conclusion, our CogTree loss is:

where λ = 1 is the balancing weight. The SGG model is trained from scratch by the CogTree loss and predicting results via flat multi-class classifier without modification.

3.3 Scene Graph GenerationModels

We case-study on three SGG models, i.e. MOTIFS, VCTree and our SG-Transformer. As shown in the bottom left of Figure 2, their architectures mainly contains two processes: object classification (bottom) for detecting the object positions and labels and relationship classification (top) for predicting the relationships between objects. We first review MOTIFS, and then introduce VCTree and SG-Transformer in a comparable fashion. All models are trained by the conventional cross-entropy loss [Zellers et al., 2018].

MOTIFS. A pre-trained Faster R-CNN [Ren et al., 2015] is applied to extract top K objects O = {oi}K and describe object oi by the RoIAlign visual feature vi, the tentative object label li, and the spatial feature bi. The above three kinds of features are concatenated and fed into Encodero to obtain the refined object embedding mi. mi is then decoded by Decodero to predict its final object label lo i . MOTIFS implemented Encodero as a Bi-LSTM and adopts an LSTM as Decodero. The relationship encoder Encoderr first adopts another Bi-LSTM to capture the contextual information and then fuses the pairwise object features with the union region feature for decoding. The relationship decoder Decoderr is a fully connected layer followed by a softmax layer. VCTree. Different fromMOTIFS, VCTree utilizes two independent TreeLSTMs as Encodero and Encoderr, respectively. It additionally takes the bounding box pair feature as another input of Decoderr.

SG-Transformer. It adopts the transformer architecture as Encodero and Encoderr by adjusting inputs for the SGG scenario. As shown in Figure 4, Encodero contains N object-to-object (O2O) transformer blocks with the input of object embeddings {vi}K . Encoderr consists of M relationto-object (R2O) transformer blocks with the input of relationship emebddings {rij}K×K and the object embeddings {mi}K . rij is a combination of three kinds of features from objects oi and oj: spatial features, union region feature, word embeddings [Pennington et al., 2014] of the predicted object labels. The output from Encoderr is concatenated with mi and mj for decoding. Decodero and Decoderr contain a fully connected layer followed by a softmax layer.

4 Experiments

Dataset. We evaluate our models on the widely compared Visual Genome (VG) split [Xu et al., 2017], with the 150 most frequent object classes and 50 most frequent relationship classes in VG [Krishna et al., 2017]. The VG split only contains training set and test set and we follow [Zellers et al., 2018] to sample a 5K validation set from the training set.

Tasks and Evaluation. The SGG task contains three sub-tasks [Zellers et al., 2018]: (1) Predicate Classification (PredCls) takes the ground-truth object labels and bounding boxes for relationship prediction; (2) Scene Graph Classification (SGCls) takes ground-truth bounding boxes for object and relationship prediction; (3) Scene Graph Detection (SGDet) predict SGs from scratch. The conventional metric Recall@K (R@K) is reported but not considered as the main evaluation due to its bias. We adopt the recently proposed balanced metric mean Recall@K (mR@K) [Chen et al., 2019b; Tang et al., 2019]. mR@K calculates R@K for each class independently, and then average the results.

Implementation. The object detector is the pre-trained Faster R-CNN [Ren et al., 2015] with ResNeXt-101-FPN [Lin et al., 2017a]. λ is set to 1 and β is set to 0.999. SGTransformer contains 3 O2O blocks and 2 R2O blocks with 12 attention heads. Models are trained by SGD optimizer with 5 epochs. The mini-batch size is 12 and the learning rate is 1.2×10-3 . Experiments are implemented with PyTorch and conducted with NVIDIA Tesla V100 GPUs.

4.1 State-of-the-Art Comparison

We evaluate the CogTree loss on three models: MOTIFS, VCTree and SG-Transformer, and compare the performance with the state-of-the-art debiasing approach TDE [Tang et al., 2020]. We also compare with the biased models, including IMP+ [Xu et al., 2017], FREQ [Zellers et al., 2018], KERN [Chen et al., 2019b]. The results are shown in Table 1. In the view of mR@K, CogTree remarkably improves all the baselines on all the metrics. The strongest baseline VCTree also achieves 3.6%∼13.6% boost. CogTree consistently outperforms conventional debiasing methods: focal loss, reweight and resample, and the SOTA debiasing method TDE. Wherein, mR@20 achieves superior improvement compared with mR@50/100, which indicates that more accurate predictions have higher rank by CogTree. It is worth highlighting that CogTree outperforms TDE on both mR@K and most metrics of R@K on the strongest baseline VCTree.

Analysis of Labeling Bias. We observe a drop of R@K with the increase of mR@K from baselines to CogTree. Further analysis in Figure 5 reveals that annotators prefer simple and vague labels, which form the head classes. The baselines are biased on these head classes and result in high R@K. CogTree prefers fine-grained and semantic-rich tail labels, causing the low R@K by "incorrectly" classifying the head classes into reasonable tail ones. To fairly compare with baselines on the conventional metric R@K, we compute R@K of the tail 45 classes from the predictions of all the classes. We observe a significant improvement on R@K from baselines to CogTree in Table 2, which proves the strong debiasing ability of CogTree without sacrificing the performance of R@K.

4.2 Ablation Study

We investigate each loss terms, the weighting method, two tree building strategies and two methods for computing PCP

and CBW: 1) LTCB: tree-based classficition loss term in Eq. 3; 2) LCB: flat classification loss term in Eq. 4; 3) LTCE: we remove the weights {wi} from Eq. 3; 4) LCE: we remove the weight wk from Eq. 4; 5) Fuse-subtree: we mix relationships belonging to different concepts in one subtree; 6) Fuse-layer: we mix the fine-grained and coarse relationships in the same layer of each subtree; 7) L(MAX): we use MAX instead of AVERAGE to compute zi and wi for non-leaf nodes in Eq.1 and Eq.2; 8) L(SUM): we use SUM instead of AVERAGE.

The results in Table 3 on the SG-Transformer baseline indicate that: 1) LTCB and LCB have complementary benefits for SGG, where LTCB has obvious greater influence. 2) Both of the loss terms benefit from the weighting strategy. 3) The CogTree is built based on two principles: a) Relationships belonging to the same concept are organized in one subtree and b) relationships in one subtree are organized in different layers from coarse to fine. A significant performance decrease is observed when the tree violates either of the principles. 4) AVERAGE performs the best because that both MAX and SUM increase the predicted probabilities and decrease the incorrect prediction penalty in Eq. 3.

4.3 Qualitative Analysis

We visualize several PredCls samples in Figure 6. SGTransformer achieves obvious improvement when equipped with CogTree: (1) CogTree predicts more fine-grained relationships. As shown in the first two rows, the baseline mostly predicts the vague head classes on and near while CogTree predicts actions and locations like parked on and in front of. (2) CogTree accurately distinguishes visually and semantically similar relationships as shown in the last row. However, the baseline falsely predicts walking on as standing on and predicts in front of as behind, since flat classification cannot capture the detailed discrepancy. In summary, CogTree preferences towards fine-grained relationships instead of the vaguely biased ones.

4.4 Parameter Analysis

In Table 4, we assess the effect of λ in Eq.5. λ = 1 achieves the best performance on most metrics. The performance drops slightly in the range of [0.7, 1.3]. We also vary the number of O2O and R2O blocks from 2 to 4 in SG-Transformer and obtain the highest mR@K with 3 O2O blocks and 2 R2O blocks. We use the above settings in our full model.

5 Conclusion

In this paper, we propose a CogTree loss to generate unbiased scene graphs with highly biased data. We novelly leverage the biased prediction from SGG models to organize the independent relationships by a tree structure, which contains multiple layers corresponding to the relationships from coarse to fine. We propose a CogTree loss specially for the above tree structure that supports hierarchical distinction for the correct relationships while progressively eliminating the irrelevant ones. The loss is model-agnostic and consistently boosting the performance of various SGG models with remarkable improvement. How to incorporate commonsense knowledge to optimize the CogTree structure will be our future work.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 62006222).