Regression Residual Reasoning with Pseudo-labeled Contrastive Learning for Uncovering Multiple Complex Compositional Relations

1 Introduction

Visual recognition tasks [Li et al., 2022; Jiang et al., 2022; Ding et al., 2022; Zhang et al., 2023; Zhang et al., 2024] focus on categorizing images into classes, while humans can not only understand the appearance but also reason the underlying rules for real-world objects [Song et al., 2023; Małki´nski and Ma´ndziuk, 2023]. Abstract Visual Reasoning

∗Corresponding author.

(AVR) has emerged recently to research the induction, summarization, and application of visual rules, which discovers the underlying rules based on contextual information, e.g., Raven's Progressive Matrices (RPM) [Zhang et al., 2019], Odd-one-out [Zerroug et al., 2022], Abstraction and Reasoning Corpus [Chollet, 2019], etc. Most AVR methods [He et al., 2023; Xu et al., 2023; Yang et al., 2023] contain two main building blocks: a visual perception module to understand the image scenes and an analogical reasoning module to conduct logical reasoning based on the perceived visual attributes.

AVR tasks were initially constructed using simple regular shapes such as polygons and lines to assess human's abstract visual reasoning [Zhang et al., 2019]. The underlying rules are intentionally made simple so that humans can discover the rules from limited images, e.g., a question panel of 3 × 3

images in RAVEN [Zhang et al., 2019]. In real applications, the patterns we humans comprehend often arise from intricate compositions of multiple rules on complex scenes. Recently, Compositional Visual Relation (CVR) [Zerroug et al., 2022] has been developed to simulate diverse compositional relations in the real world, in which 103 compositional rules are much more comprehensive and complex than rules in other datasets [Zhang et al., 2019]. Furthermore, instead of regular shapes, the CVR dataset is composed of irregular curved shapes, posing additional challenges for visual perception.

The emergence of CVR datasets partially solves the problems in AVR tasks, but there are still some challenges. 1) As most existing datasets are composed of simple rules or regular shapes, the model may complete the tasks via surface matching of image appearance, instead of uncovering the underlying rules. 2) The irregular curved shapes in CVR [Zerroug et al., 2022] and the regular shapes in RAVEN [Zhang et al., 2019] and PGM [Barrett et al., 2018] have limited variations in attributes, which not only leads to the problem of appearance matching but also artificially simplifies the tasks. 3) The problem formulation of AVR tasks is relatively simple, e.g., the 3 × 3 question panel in RAVEN [Zhang et al., 2019] and one outlier out of four images in CVR [Zerroug et al., 2022].

To tackle these problems, a new AVR task of Multiple Complex Compositional Reasoning (MC2R) is proposed. As shown in Figure 1, given three images following a compositional rule, and two outliers following slightly different compositional rules, the target is to identify the two outliers. The proposed MC2R addresses the challenges of existing AVR tasks in three aspects. 1) Outliers look similar to normal images as the underlying rules only differ slightly, which avoids the problem of potential shortcuts in appearance matching, and enforces AVR models to focus on the subtle rule differences. 2) The attribute variations across images are intentionally increased, to avoid possible appearance matching and greatly challenge identifying outliers. 3) Compared to identifying one outlier out of four images in CVR [Zerroug et al., 2022], the proposed MC2R task is much more challenging, requiring identifying two outliers from five images. As shown in experiments, the MC2R task greatly challenges existing AVR models and pushes the frontier of AVR research.

To effectively detect the subtle rule differences, Regression Residual Reasoning with Pseudo-labeled Contrastive Learning (R3PCL) is proposed, containing a Regression Residual Reasoning Module (R3M) for logic reasoning and a backbone of ResNet-50 with Pseudo-labeled Contrastive Learning (PCL) for visual perception. Due to the tiny differences in compositional rules, the limited number of images per question panel, and the large attribute variations, it is difficult to directly identify the outliers. As shown in Figure 1, we transform the original problem of identify-2out-5 into the problem of determining the only selection of three normal images from C 3 5 = 10 possible selections. The key motivation here is that by selecting the three images that follow the same compositional rule, we can regress any one image with minimal error by using the other two, while the regression error for other selections containing outliers will be larger. Inspired by this, a Regression Residual Reasoning Block (R3B) is proposed, where one image is regressed using the other two through a series of regression networks, and the regression error is minimal when three images follow the same compositional rule. Then, a set of R3Bs are cascaded to form the proposed R3M. As a result, the proposed method could better capture the tiny differences in compositional rules and more accurately identify the outliers.

Lastly, the increased attribute variations across images and the tiny rule differences pose great challenges to visual perception. To tackle the distribution discrepancy between training and test data due to large sample variations, the proposed PCL incorporates a data augmentation of image rotation of 90◦ , 180◦ , and 270◦ to improve the generalizability of the extracted features. More importantly, the proposed PCL incorporates a contrastive learning strategy to better distinguish normal images and outliers. After pseudo-labeling the five images, the more representative outlier is selected through a set of min-max operations, and the most representative pair of normal images is identified as the most similar pair. The proposed PCL then simultaneously maximizes the similarity between the pair of normal images and minimizes the similarity between the outlier and the normal images. After training converges, the proposed PCL improves the model in handling subtle rule differences between normal images and outliers.

Our contributions can be summarized as follows. 1) To avoid possible shortcuts in appearance matching, an MC2R task with a benchmark dataset is proposed to challenge existing AVR models in comprehending complex visual compositional relations. 2) To better detect the tiny rule differences, the proposed model transforms the MC2R task into a select-1out-10 problem, and designs a Regression Residual Reasoning module to determine the three images that follow the same compositional rule. 3) To boost the generalizability, the proposed visual perception module incorporates a data augmentation mechanism into the contrastive learning framework, which contrasts the pseudo-labeled normal images and outliers to enhance the discriminant power of the model. 4) Extensive experimental results show that the proposed MC2R dataset is much more challenging than existing benchmark datasets, and the proposed method significantly outperforms state-of-the-art solutions on two benchmark datasets.

2 RelatedWork

Abstract Visual Reasoning. Abstract Visual Reasoning explores the capacity to draw analogies regarding abstract visual relations across diverse scenes. It is often used to evaluate human's intelligence in applying relevant knowledge and past experiences in unfamiliar contexts [Małki´nski and Ma´ndziuk, 2023]. AVR contains several tasks: Raven's Progressive Matrices (RPM) [Zhang et al., 2019; Barrett et al., 2018], Odd-one-out [Zerroug et al., 2022], Abstraction and Reasoning Corpus (ARC) [Chollet, 2019], etc. RAVEN [Zhang et al., 2019] and PGM [Barrett et al., 2018] are typical RPM benchmarks, where the question panel involves a 3 × 3 grid with 8 contextual images and a missing one, and the task is to identify the missing image from 8 candidates. The rules in RPMs are often too simple and fail to reflect the complexity of compositions. Odd-one-out is better suited for creating complex compositional rules [Zerroug et al., 2022], where the task is to identify the outlier from four given images. ARC derives the masked block based on the example grid-like color blocks [Chollet, 2019]. As many AVR tasks were initially designed to evaluate human's abstract reasoning, they are intentionally composed of simple patterns following simple rules.

Compositionality assists humans in gaining a deeper understanding of the essence of objects [Hofstadter, 2001]. It has been explored in diverse fields, including logical reasoning [Lake and Baroni, 2018], visual reasoning [Johnson et al., 2017; Thrush et al., 2022; Zerroug et al., 2022], and mathematics [Saxton et al., 2018]. Typically, the CVR dataset [Zerroug et al., 2022] has been created to evaluate the reasoning of compositionality, which applies compositional relations to a set of attributes and generates a wide range of tasks that adhere to compositional rules. In this paper, the proposed MC2R task addresses the limitations of existing AVR tasks, typically CVR tasks, and challenges existing solution models.

Many methods have been developed to solve AVR tasks, which extract visual attributes using a perception module and exploit their relations using a reasoning module [Barrett et al., 2018; Wu et al., 2020; He et al., 2023; Yang et al., 2023]. As early AVR tasks are composed of simple regular shapes, shallow networks such as ResNet-18 [Hu et al., 2021] and ResNet-50 [Barrett et al., 2018; Zhang et al., 2019] are often utilized for visual perception. Yang et al. [2023] argued that deep neural networks are not suitable for AVR problems where the objects are relatively small, so they adopted four residual blocks for visual perception. He et al. [2023] argued that local attributes such as shape, size and color can be well detected by shallow networks, while global attributes such as number and position may be better detected using deep networks, so they developed Hierarchical ConViT to perceive the visual attributes at different receptive fields.

To deduce the relations between attributes, early models [Barrett et al., 2018] often utilize multi-layer perceptrons or ResNet blocks. Recently, several specialized reasoning blocks have been developed. Barrett [Barrett et al., 2018] first encoded context-context relations and context-multiplechoice relations based on extracted visual features, and iteratively applied Relation Networks to deduce inter-panel relations. Wu et al. [2020] introduced the Scattering Compositional Learner (SCL) for feature extraction through object networks and attribute networks, and designed a similar SCL for abstract reasoning. Yang et al. [2023] developed PredRNet to deduce abstract rules by directly mapping from the eight context images to the missing image. Very recently, He et al. [2023] developed an attention-based relational reasoner to discern the underlying relations. Despite these recent advancements, there may still exist shortcuts for appearance matching due to the nature of existing AVR tasks, which hinders understanding of underlying rules.

Contrastive Learning. Contrastive learning maintains representation consistency across various data views while enhancing discrimination between different data points [Son, 2022]. Related research mainly focuses on two directions. 1) The design of training strategies for enhancing the generalization capabilities of a backbone [Chen et al., 2020;

Chen et al., 2021]. SimCLR [Chen et al., 2020] preserves the similarity between a sample and its augmented versions, and ensures their dissimilarity from other samples, with a learnable nonlinear transformation to enhance the feature representation. Chen et al. [2021] developed the widely adopted MoCo (Momentum Contrast), utilizing a queue to augment negative samples and a momentum encoder to maintain the feature consistency. 2) The formulation of the loss function to enhance the discrimination among samples [Chuang et al., 2022]. Specifically, the InfoNCE loss [Oord et al., 2018] follows the structure of BCE loss but incorporates a hyperparameter to address the challenge of identifying dissimilar negative samples. Robinson et al. [2020] devised an adjustable sample distribution to highlight challenging negative samples. Chuang et al. [2022] presented RINCE to enforce the symmetry property in the loss function, thereby enhancing the robustness of the derived features. The proposed PCL exploits pseudo labels and contrastive learning with data augmentation to extract generalized high-level abstract features that better distinguish normal images and outliers.

3 ProposedMethodology

3.1 ProblemFormulation

As AVR tasks are often constructed using simple rules on regular shapes [Zhang et al., 2019], the solution may be derived from appearance matching instead of uncovering the underlying rules. In addition, most existing problem formulations are single-answer questions, e.g., identifying one outlier from four images in Odd-one-out [Zerroug et al., 2022] or selecting the most suitable option to complete a 3×3 question panel in RPMs. Furthermore, image variations are often limited due to simple regular shapes in AVR datasets. All these simplify the tasks and artificially boost the performance of AVR models.

To address these issues, a more challenging AVR task of Multiple Complex Compositional Reasoning (MC2R) is proposed. As shown in Figure 1, given a question panel of five images, {Xi ∈ RH×W×C }5i=1 , three are generated from the same compositional rule formed by applying specific relations on random object attributes in a tree-like manner and two are generated as outliers by applying slightly different compositional rules on the same attributes, the task is to identify the two outliers, where H, W, and C are the height, width, and number of channels of images, respectively.

The MC2R tasks well address the issues of existing AVR datasets. Firstly, each question panel is constructed using compositional rules rather than simple rules, and the differences in compositional rules between outliers and normal images are intentionally kept minimal. Secondly, instead of regular shapes, curved contours are utilized to increase variations in image appearance. Compared to the CVR dataset [Zerroug et al., 2022], the attribute variations are intentionally randomly sampled from a much larger range, bringing significant appearance differences between images generated from the same rule. Lastly, as the proposed MC2R identifies two outliers from five given images, this multi-answer question is much more challenging than the single-answer questions in PGM [Barrett et al., 2018], RAVEN [Zhang et al., 2019], and CVR [Zerroug et al., 2022]. All these measures prevent the model from completing tasks through appearance matching and greatly increase the difficulty of the MC2R task.

Given the input Xi with the label yi = 1 if Xi is an outlier and 0 otherwise, the training process can be represented as,

where ˆ y is the one-hot encoding of regressed labels, Θ denotes the model parameters, and FΘ is the mapping function.

3.2 Overview of Proposed R3PCL

To solve MC2R tasks, Regression Residual Reasoning with Pseudo-labeled Contrastive Learning (R3PCL) is proposed, which contains two main modules. 1) The visual perception module consisting of an image encoder with a pre-trained ResNet-50 [Chen et al., 2021] as the backbone, and a Pseudolabeled Contrastive Learning (PCL) block to boost the feature representations. In contrast to unsupervised learning in most contrastive learning models [Khosla et al., 2020], the proposed PCL pseudo-labels normal images and outliers, and subsequently designs a set of min-max similarity pooling operations to select the more representative outlier and a max-pooling operation to select the pair of most similar normal images. Then, to enhance the feature representation, contrastive learning with data augmentation is designed to minimize the similarity between the outlier and normal images and maximize the similarity between normal images. 2) Regression Residual Reasoning Module (R3M), which converts the identify-2-out-5 formulation into the select-1-out-10 formulation. Then, a set of Regression Residual Reasoning Blocks (R3Bs) are designed to identify the triplet of three normal images, where one image can be regressed by the other two with minimal error. The proposed R3M detects subtle differences in underlying composition rules by minimizing the regression error among three normal images, and hence more accurately identifies them and, equivalently, the two outliers.

3.3 Pseudo-labeled Contrastive Learning

The proposed PCL promotes similarity among images adhering to the same compositional rule while discouraging similarity among images following different rules. To achieve this, we leverage the initial prediction results of R3M to pseudo-label normal images and outliers. Denote {Ni}3i=1 and {Oj}2j=1 as the features for pseudo-labeled normal images and outliers respectively. These pseudo-labels may contain errors and hence directly employing them introduces noise. To tackle this problem, similarity pooling operations are designed to improve the quality of pseudo-labels, where the cosine similarity, S(Fi,Fj) = Fi×Fj |Fi|×|Fj| , is used. It can be observed that normal images following the same compositional rule exhibit higher similarity than images following different rules. We hence select the pair of normal images that exhibit the largest feature similarity,

where Fmax(·) is the max-pooling operation. Denote N c 1 and N c 2 as the two normal images with the highest confidence.

Next, we select one outlier with higher confidence from two pseudo-labeled outliers. Intuitively, outliers are least similar to the normal images. We hence employ min-maxpooling to identify the outlier. Specifically, we first derive the nearest neighbor for each pseudo-labeled outlier by the maxpooling operation, and then choose the one with the least similarity as the outlier by the min-pooling operation Fmin(·),

Denote Oc as the outlier. The PCL loss LC is defined as,

3.4 Regression Residual ReasoningModule

Challenges of MC2R. There are many challenges in solving the MC2R tasks. 1) For each question panel, there are only three normal images and two outliers, so it is inherently challenging to uncover the rules from such limited images. In addition, there are two correct outliers for each question panel, while most existing AVR tasks contain single-answer questions. 2) The compositional rules for normal images and outliers only differ slightly, so their appearance differences are challenging to observe, while the attribute variations across images are intentionally increased to avoid appearance matching. 3) The MC2R tasks contain as many as 103 diverse compositional rules, compared to several simple rules in many AVR tasks. Furthermore, they are composed of levels of compositions, in contrast to existing datasets without rule compositions [Barrett et al., 2018; Zhang et al., 2019]. To tackle the challenges, the proposed R3M transforms the identify-2-out-5 problem into a select-1out-10 problem, and then designs a set of R3Bs to identify the three normal images, and hence identify the two outliers.

ProblemTransformation. It is hard to directly identify the two outliers from five given images. Hence, the multi-answer task is transformed into a single-answer problem of select-1out-10. Specifically, given five images, there are C 3 5 = 10 possible ways to select three images, and only one consists of three normal images, i.e., {F s i }3i=1 = Fse({Fj}5j=1 ), where Fse(·) denotes the function to select three images {F s i }3i=1 from five given images {Fj}5j=1 . If three normal images are selected, one image can be well regressed by the other two with minimal error, while the error will be large if there is one or two outliers in these three images. By identifying the image triplet with the smallest regression error E({F s i }3i=1 ), the three normal images will be accurately identified. Other formulations such as selecting more than three images will include outliers and selecting fewer than three may result in insufficient information for accurate regression.

Regression Residual Reasoning Block. To capture the subtle differences in diverse compositional rules between normal images and outliers using limited images, a set of Regression Residual Reasoning Blocks is designed for abstract reasoning. As shown in Figure 2, given three images, two of them are treated as the context images to regress the remaining one,

where Fr is the regression function. We then calculate the regression error ˜ F s 3 = F s 3 - ˆ F s 3 between the regression ˆ F s 3 and the target, concatenate ˜ F s 3 with the two context features to form ˆY = [F s 1 ,F s 2 , ˜ F s 3 ], and feed ˆY into two convolutional layers to enhance the interactions within these features, ˜Y = FC( ˆY), where FC represents the operations of two convolutional layers. Moreover, we design a residual structure to shortcut the original features Y0 = [F s 1 ,F s 2 ,F s 3 ] as,

Iterative Regression Residual Reasoning. There are multiple compositions in each rule and a single R3B is insufficient for adequate reasoning. The proposed iterative regression residual reasoning hence hierarchically combines K R3Bs,

where F kR3B denotes the reasoning block in the k-th iteration. This simulates the iterative reasoning of humans. When faced with an unfamiliar rule, humans engage in a continuous process of guessing, validating, and ultimately confirming the rule, corresponding to the iterative reasoning process in R3M.

Loss Function forR3M. We convert the label of each question panel into a one-hot vector y = {yi}10 i=1 , where yi = 1 if three normal images are selected, and 0 otherwise. Following previous AVR models [Benny et al., 2021], two fully connected layers are designed as the classifier Fϕ(·), to determine which one among 10 selections consists of three normal images. The Binary Cross Entropy Loss is used in this paper,

where LR denotes the regression loss, and σ(·) denotes the sigmoid function. The final loss for our end-to-end model is,

where λ is the weighting factor to balance the two losses.

4 Construction of MC2RDataset

The MC2R dataset contains 9 elementary tasks involving one relation for each task, and 94 compositional tasks combining two or more relations for each task, i.e., 20 using single relations, 65 using a pair of relations, and 9 using more than two relations. For each task, 10,000, 500 and 1,000 samples are generated for training, validation and testing, respectively.

Each image consists of three elements. 1) Object, a closed irregular contour with 9 fundamental attributes: shape, position, color, size, rotation angle, and flip state of an object, inside representing that one object contains another, contact that two objects are contacting, and count representing the number of objects. 2) Relation, defined on attribute as: logical operators such as and ∧, or ∨, not ¬; relational operators such as greater than >, less than <, equal to =; arithmetic operators such as plus +, minus -. 3) Rule. As shown in Figure 1, objects are treated as bottom-level nodes, relations are treated as intermediate and upper-level nodes, and a rule is composed by a hierarchical composition of nodes in a tree-like manner.

Each MC2R task is created with three images following the same compositional rule, and two outliers following slightly different rules, e.g., by changing a relation node in the rule from = to >. The rule differences are intentionally minimized to challenge AVR models and avoid appearance matching. The variations of attributes that are not used in the compositional rules are also intentionally increased, e.g., if the compositional rule includes rotation, the shape and color of objects, which are unrelated to the compositional rule, are kept as diverse as possible. Compared to singleanswer questions in CVR, it is much more challenging to reason the multiple answers on the MC2R dataset, on top of the increased attribute variations. Compared to simple rules on regular shapes in PGM [Barrett et al., 2018] and RAVEN [Zhang et al., 2019], the proposed MC2R is constructed using much more complex and diverse compositional rules on irregular contours, and hence more challenging.

5 Experimental Results

5.1 Experimental Settings

The proposed R3PCL is compared with three state-of-the-art models in solving RPMs but adapted for solving CVR and MC2R problems, i.e., WReN [Barrett et al., 2018] utilizing a Relation Network to infer inter-feature relations for RPM problems; SCL [Wu et al., 2020] containing an object network for encoding objects, an attribute network for encoding attributes, and a relation network for unveiling inherent relations; PredRNet [Yang et al., 2023] comprising an image encoder with four residual blocks to extract visual features and a predictive reasoning module to capture the relations between contextual images and candidate answers.

R3PCL is also compared with eight state-of-the-art models for solving CVR problems, i.e., ResNet-50 [He et al., 2016] and ViT-small [Dosovitskiy et al., 2020] are utilized as the backbone image encoder, and incorporated with two fully connected layers as the reasoning module, same as in [Zerroug et al., 2022]. IN-ResNet-50 [He et al., 2016] and In-ViT-small [Dosovitskiy et al., 2020] pre-trained on ImageNet [Deng et al., 2009], CLIP-ResNet-50 and CLIPViT-base employing the visual encoder of CLIP [Radford et al., 2021], and SSL-ResNet-50 and SSL-ViT-small utilizing MoCo-v3 [Chen et al., 2021] to pre-train the backbone with 1.03 million samples are chosen for comparisons, and the remaining configurations are consistent with ResNet-50 [He et al., 2016] and ViT-small [Dosovitskiy et al., 2020].

The input image dimensions are 128×128×3 pixels. The number of R3Bs is set to K = 3 and the weighting factor λ is set to 1, as indicated by the ablation study. The Adam optimizer is employed with an initial learning rate of 1×10-4 and a weight decay of 1 × 10-4 . The batch size is set to 64.

The maximum number of epochs is set to 100, and training stops if there is no substantial performance gain in 10 epochs.

5.2 Comparisons on CVRDataset

Extensive comparison experiments are conducted on the CVR dataset, following the same settings as in [Zerroug et al., 2022], by reporting the reasoning accuracy for 20, 50, 100, 200, 500, 1,000, and 10,000 training samples per task. As shown in Table 1, the proposed R3PCL significantly and consistently outperforms all the compared models under all the settings. Specifically, compared to the second-best methods, SSL-ResNet-50 [Chen et al., 2021], the performance gains using 500 and 1,000, and 10,000 training samples are 7.5%, 9.7% and 2.6%, respectively, showing the superior ability of R3PCL in solving compositional reasoning problems.

Figure 3 presents the accuracy of SSL-ResNet-50 and R3PCL in reasoning pairwise compositional rules using 1,000 training samples per task on the CVR dataset. Our method consistently outperforms SSL-ResNet-50 for all pairs of elementary rules except those where both methods perform perfectly. In particular, the proposed R3PCL presents almost perfect results on the diagonal representing the accuracy on elementary rules and their compositions, while SSL-ResNet-50 performs poorly, especially rules related to rotation and count that require high-level image semantics.

5.3 Comparisons onMC2RDataset

Extensive experiments are conducted on the MC2R dataset, following the same settings as in [Zerroug et al., 2022]. From Table 1, the following can be observed. 1) The proposed R3PCL outperforms all the compared methods for all the settings, demonstrating its ability to distinguish the subtle differences in compositional rules despite the challenges of the MC2R dataset. 2) When the number of training samples per task is less than 200, many methods have very low accuracy, even approaching a random guessing rate of 10%, while our method achieves a higher accuracy of 19.8%, 27.8%, 35.1% and 45.5% using 20, 50, 100, and 200 training samples, respectively. This not only indicates that our MC2R dataset is more challenging, but also demonstrates the stronger reasoning abilities of our R3PCL. 3) Compared to the second-best method, SSL-ResNet-50, the performance gain of R3PCL grows as the number of training samples increases, as more samples assist the model to better capture visual information and compositional rules. Even when there are 10,000 training samples per task and the benefits of contrastive learning with data augmentation are reduced, the proposed R3PCL still yields a large performance gain of 5.6%.

Overall, all the evaluated methods perform poorer on the MC2R dataset than on the CVR dataset, showing that the MC2R is more challenging. Even so, the proposed R3PCL significantly outperforms all the compared methods on both datasets, demonstrating its superior ability in reasoning compositional rules.

Figure 3 shows the reasoning accuracy of SSL-ResNet-50 and R3PCL for individual tasks on the MC2R dataset. The proposed method significantly outperforms SSL-ResNet-50 for all pairs of elementary rules. While SSL-ResNet-50 struggles with rules related to rotation and flip, the proposed R3PCL performs much better on these tasks, demonstrating its effectiveness in capturing high-level image semantics and uncovering the subtle rule differences.

5.4 Ablation Studies

We ablate the two major components of R3PCL on the MC2R dataset. SSL-ResNet-50 [Chen et al., 2021] serves as the baseline, and its perception module and reasoning module are replaced by PCL and R3M respectively. As shown in Table 2, the proposed PCL achieves a performance gain of 2.3%, 4.0%, and 12.9% for 200, 500, and 1,000 training samples per task respectively, indicating that it could better distinguish normal images and outliers through contrastive learning. The performance gains brought by R3M are 11.7%, 13.5%, and 14.0% respectively, proving that R3M could better capture the subtle rule differences and identify outliers. By adopting PCL and R3M simultaneously, the accuracy is further boosted to 45.5%, 59.6%, and 72.9%, respectively. The ablation results demonstrate the effectiveness of the two proposed modules.

Our model is ablated using different numbers of R3Bs with 1,000 training samples per task on the MC2R dataset. As shown in Table 3, when K is increased from 1 to 3, there is a noticeable performance improvement from 71.2% to 72.9%, indicating that iterative reasoning better deduces complex compositional rules. The performance at K = 4 drops to 70.4%, due to possible over-fitting by using too many R3Bs.

Lastly, we evaluate the weighting factor λ in Equation (9). As shown in Table 3, for λ = 0.25 or 0.5, the weight of LC is too small, causing the perception module to insufficiently extract discriminative and generalized features. When λ = 2, the accuracy drops slightly compared to λ = 1 due to overemphasis on contrastive learning. Hence, λ is set to 1 by default.

6 Conclusion

It is challenging to reason over compositional rules on the CVR dataset, but the single-answer question formulation and limited attribute variations potentially allow models to complete tasks through appearance matching. To address this problem, an MC2R benchmark is developed to challenge compositional reasoning, which has increased attribute variations, smaller rule differences between normal images and outliers, and multiple correct answers. The proposed R3PCL well tackles the challenges of MC2R tasks by enhancing feature representations through Pseudo-labeled Contrastive Learning and strategically transforming the original problem into a select-1-out-10 problem, and effectively solves the task through iterative regression residual reasoning. The experimental results on the CVR and MC2R datasets demonstrate that the proposed R3PCL significantly and consistently outperforms all the compared models under various settings. Code is available here.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 72071116, and in part by the Ningbo Science and Technology Bureau under Grants 2022Z173, 2022Z217, 2023Z138 and 2023Z237.

Contribution Statement

The contributions of Chengtai Li and Yuting He to this paper were equal.