A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning

1 Introduction

Generalization, the ability of a model to perform well on unseen data, remains a fundamental challenge in deep learning (DL). While DL methods have demonstrated remarkable achievements in various domains, their generalization capabilities are often questioned, particularly in tasks that demand abstract problem-solving and reasoning skills [Chollet, 2019]. One such domain is abstract visual reasoning (AVR) [Mitchell, 2021; Stabinger et al., 2021; van der Maas et al., 2021; Małki´nski and Ma´ndziuk, 2023] that encompasses tasks requiring (human) fluid intelligence - an aspect of human cognition believed to be crucial for reasoning in never-encountered settings [Carpenter et al., 1990]. The most popular AVR tasks are Raven's Progressive Matrices (RPMs) [Raven, 1936; Raven and Court, 1998], which constitute a common problem found in human IQ tests. Typical RPMs comprise two components - the context panels arranged in a 3×3 grid with the bottom-right panel missing and up to 8 answer panels, out of which only one correctly completes the matrix. Solving an RPM instance requires identification of underlying abstract rules applied to certain attributes of the objects composing the instance (see Fig. 1 for an illustrative example).

Design of computational methods capable of tackling RPMs has for decades been an active area of research [Evans, 1964; Foundalis, 2006; Lovett et al., 2007; Kunda et al., 2010]. Consequently, a number of works considered automatic creation of RPM datasets [Matzen et al., 2010; Wang and Su, 2015; Ma´ndziuk and ˙Zychowski, 2019] and a wide suite of predictive models [Hern´andez-Orallo et al., 2016; Hern´andez-Orallo, 2017] were proposed, with DL methods showing the most promising performance [Yang et al., 2022; Małki´nski and Ma´ndziuk, 2025b]. While this rapid progress led to exceeding the human level in particular problem setups [Wu et al., 2020; Mondal et al., 2023], a fundamental challenge of generalization to novel problem settings remains largely unattained.

Initial works designed several RPM datasets [Matzen et al., 2010; Wang and Su, 2015; Hoshen and Werman, 2017], however, measuring generalization was not their focus. While some works explored knowledge transfer between related tasks [Ma´ndziuk and ˙Zychowski, 2019; Tomaszewska et al.,

2022], the complexity of the datasets was limited and consequently they didn't pose a challenge for contemporary DL methods. To measure generalization in modern DL models, the PGM dataset was introduced [Barrett et al., 2018]. PGM defines eight generalization regimes, each specifying the distribution of objects, rules and attributes in train and test splits. For instance, in the Held-out Triples split, a given rule-object-attribute triplet (e.g. Progression on Object's Size) was assigned only to one of the two splits. In effect, the models were tested on triplet combinations different from training ones, allowing to assess their generalization capabilities. A subsequent work proposed RAVEN [Zhang et al., 2019a], another RPM dataset with enriched perceptual complexity of matrices instantiated in seven visual configurations (Center, 2x2Grid, 3x3Grid, Left-Right, Up-Down, Out-InCenter, Out-InGrid). Moreover, the benchmark is characterized by a moderate sample size, i.e. 70K instances, compared to 1.42MRPMs per each of the eight regimes in PGM. Due to this size disparity, subsequent research gravitated towards RAVEN and its revised variants (I-RAVEN [Hu et al., 2021] and RAVEN-Fair [Benny et al., 2021]), which didn't require substantial computational resources to train DL models.

Contribution. Drawing inspiration from the broad adoption of RAVEN and the generalization assessment capabilities of PGM, this paper proposes a novel suite of generalization challenges stemming from I-RAVEN [Hu et al., 2021] (a revised variant of RAVEN that removes a bias in RAVEN's answer panels). However, unlike I-RAVEN, the proposed suite of benchmarks allows for a direct assessment of the generalization and knowledge transfer of AVR models. Compared to PGM, our datasets feature compositionality and variety of figure configurations, and their processing doesn't require substantial computational resources. Furthermore, they include structural annotations, which are utilized, for example, in recent neuro-symbolic approaches [Zhang et al., 2021; Zhang et al., 2022].

First, we introduce Attributeless-I-RAVEN (A-I-RAVEN), comprising 10 generalization regimes. The 4 primary regimes correspond to specific held-out attributes ({Position, Type, Size, Color}), resp. The training matrices in these regimes adhere to the Constant rule for the respective attribute, whereas test matrices employ a rule different from Constant for this attribute (i.e., Progression, Arithmetic, or Distribute Three). Moreover, we propose 6 extended regimes: 3 of them feature a held-out attribute pair, while another 3 replace the Constant rule in the training set with each remaining rule. In effect, each regime comprises different distributions of training and test data.

Next, we propose I-RAVEN-Mesh, a variant of I-RAVEN with a new grid-like structure overlaid on the matrices. The dataset enables assessing generalization to incrementally added structures and progressive knowledge acquisition in a transfer learning (TL) setting.

Investigations involving 13 contemporary AVR DL models reveal that the introduced benchmarks present a substantial challenge for the tested methods, raising the need for further advancements in this area.

The key contributions of the paper are summarized below.

• We introduce the A-I-RAVEN dataset that enables measuring generalization across 10 regimes.
• We construct I-RAVEN-Mesh, an extension of I-RAVEN with a new component structure that facilitates assessment of progressive knowledge acquisition in a TL setting.
• We evaluate the performance of state-of-the-art AVR models on the introduced benchmarks, uncovering their limitations in terms of generalization to novel problem settings.

2 RelatedWork

Generalization in AVR. In recent years, a variety of AVR problems and corresponding datasets have emerged [Nie et al., 2020; Fleuret et al., 2011; Qi et al., 2021; Shanahan et al., 2020; Hill et al., 2019; Zhang et al., 2020] and several attempts have been made to measure generalization in contemporary AVR models based on the introduced benchmarks. In particular, distinct visual configurations were employed in RAVEN to assess how a model trained on one configuration performs on the remaining ones [Zhang et al., 2019a; Spratley et al., 2020; Zhuo and Kankanhalli, 2021]. Although in such a setting the visual aspects of train/test matrices come from different distributions, the underlying rules and attributes remain the same. In contrast, A-I-RAVEN enables studying the generalization of rules applied to held-out attributes, shifting the focus from perception towards reasoning. Besides RPMs, the limits of generalization have been explored in other AVR tasks as well. Visual Analogy Extrapolation Challenge evaluates model's capacity for extrapolation [Webb et al., 2020]. However, such specialized datasets might favor models that explicitly embed the notion of extrapolation in their design and aim for being invariant only to specific attributes such as object size or location. Differently, our benchmarks allow verifying the model's capacity to learn a given concept from the data and generalize it to novel settings. This perspective links our work to the recent literature on concept learning [Moskvichev et al., 2023]. However, the concept-oriented benchmarks that originate from ARC [Chollet, 2019] remain largely unsolved by DL models and pose a significant challenge even for leading multi-modal large language models [Mitchell et al., 2023]. In contrast, both benchmarks proposed in this work are attainable by DL models, though further advances in generalization abilities of the models are necessary to consider them solved.

Model architectures. Preliminary attempts to solve RPMs with DL models involve WReN [Barrett et al., 2018] that reasons over object relations using Relation Network [Santoro et al., 2017], or SRAN [Hu et al., 2021] that relies on a hierarchical architecture with panel encoders devoted to particular image groups. A common theme enabling generalization in DL models is to explicitly identify RPM objects. To this end, RelBase [Spratley et al., 2020] employs Attend-Infer-Repeat, an unsupervised scene decomposition method, STSN [Mondal et al., 2023] utilizes Slot attention [Locatello et al., 2020]

to decompose matrix to slots containing particular objects and Temporal Context Normalization (TCN) [Webb et al., 2020] to normalize latent matrix panel representations in a task-specific context, DRNet [Zhao et al., 2024] relies on a dual-stream design, and MRNet [Benny et al., 2021] presents a multi-scale architecture. SCL [Wu et al., 2020] proposes the scattering transformation, CoPINet [Zhang et al., 2019b] and CPCNet [Yang et al., 2023b] rely on contrastive architectures, PredRNet [Yang et al., 2023a] learns to minimize the prediction error, ALANS [Zhang et al., 2021] and PrAE [Zhang et al., 2022] employ neuro-symbolic architectures, and SCAR [Małki´nski and Ma´ndziuk, 2024b] adapts its computation to the structure of the considered matrix. Despite the high variety of AVR models, experiments on the introduced benchmarks reveal their shortcomings in terms of generalization and knowledge transfer.

3 Proposed Datasets

The set of attributes in I-RAVEN is A = {Position, Number, Type, Size, Color} and the set of rules is R = {Constant, Progression, Arithmetic, Distribute Three}. For attribute a ∈ A and a dataset split s ∈ S, where S = {train, val., test}, we define the set of rules applicable to a in split s by R(a, s) ⊆ R. In I-RAVEN all rule-attribute pairs are valid in all splits:

3.1 Attributeless-I-RAVEN (A-I-RAVEN)

To probe generalization in DL models, we present AI-RAVEN, a benchmark composed of 10 generalization regimes. Example matrices are illustrated in Fig. 2, with additional samples provided in [Małki´nski and Ma´ndziuk, 2025a, Appendix A]. Each regime defines a set of held-out attributes A∗ , each with a corresponding rule r∗ (a), a ∈ A∗ . In train and validation splits, held-out attribute a ∈ A∗ is governed by r∗ (a). In the test split, a ∈ A∗ is governed by a different rule sampled from R- {r∗ (a)}. In effect, during training, the model doesn't see rule-attribute combinations required to solve test matrices. There are no rule-related constraints on the remaining attributes. In summary, we have:

We define 4 primary regimes with r∗ (a) = Constant that correspond to individual held-out attributes (|A∗ | = 1), denoted as A/<Attribute> (e.g., A/Type). Since Position and Number attributes are tightly coupled (e.g., it's impossible to increase cardinality of objects while keeping their position constant), we allocate a single generalization regime, A/Position, to cover both attributes. In addition, we define 6 extended regimes as supplementary generalization challenges. In the first group a pair of attributes is held-out in the training set, i.e. |A∗ | = 2. Specifically, we introduce 3 new regimes: A/ColorSize, A/ColorType, and A/SizeType, based on the respective attribute pairs. In the second group, Constant rule in r∗ (a) is replaced with each of the 3 remaining rules, leading to A/Color-Progression, A/Color-Arithmetic, and A/Color-DistributeThree regimes. While this modification could be applied to all the described regimes, we focus on the Color attribute due to its broad range of possible values.

3.2 I-RAVEN-Mesh

The other of the proposed benchmarks is designed to probe progressive knowledge acquisition in a TL setting. IRAVEN-Mesh extends I-RAVEN by introducing a novel visual component overlaid on top of the existing I-RAVEN components (see Fig. 3). Though the dataset can serve as a learning challenge on its own, the main motivation behind its introduction is to employ models pre-trained on I- RAVEN and fine-tune them on I-RAVEN-Mesh with a configurable train sample size, facilitating analysis of their TL performance. The mesh grid comprises from 1 to 12 lines placed in predefined locations. The set of available lines covers the inner and outer edges of a 2 × 2 grid (12 lines in total). The mesh component has two attributes: Amesh = {Number, Position}, which govern the count and location of lines, respectively. To each attribute a rule r ∈ R can be applied. Table 1 describes the effect of applying a given rule-attribute pair to the mesh component. To generate the mesh component of an I-RAVEN-Mesh matrix, we sample one of the two attributes a ∈ Amesh and a corresponding rule r ∈ R that governs its values. As the attributes often depend on each other (e.g., it's impossible to increase the number of lines while keeping their position constant), we don't constrain the value of the other attribute. The rule-attribute pairs for the base I-RAVEN components are generated in the same way as in the original dataset. To generate answers to the matrix, we follow the impartial algorithm proposed in IRAVEN [Hu et al., 2021]. In addition, each matrix contains at least one incorrect answer that differs from the correct one only in the mesh component, ensuring that the solver has to identify the correct rule governing the mesh component in order to solve the matrix. To facilitate training with an auxiliary loss, in which the model additionally predicts the representation of rules governing the matrix [Barrett et al., 2018], we extend the base set of rule annotations with ones concerning the Mesh component.

4 Experiments

We assess generalization of state-of-the-art models for solving RPMs on A-I-RAVEN and evaluate progressive knowledge acquisition on I-RAVEN-Mesh.

Experimental setup. In all experiments we use the Adam optimizer [Kingma and Ba, 2014] with β1 = 0.9, β2 = 0.999, ϵ = 10-8 and a batch size set to 128. Learning rate is initialized to 0.001 and reduced 10-fold (at most 3 times) if no progress is seen in the validation loss in 5 subsequent epochs, and training stops early in the case of 10 epochs without progress. Unless stated otherwise, each model configuration was trained 3 times with a different seed, and we report mean and standard deviation for these runs. In each experiment, we utilize 42000 training, 14000 validation, and 14000 test matrices, following the standard data split protocol taken in prior works [Zhang et al., 2019a; Hu et al., 2021]. All models are trained with the auxiliary loss with sparse encoding [Małki´nski and Ma´ndziuk, 2024a] and β = 1. Experiments were run on a worker with a single NVIDIA DGX A100 GPU.

Models. In addition to the simple CNN-LSTM baseline [Barrett et al., 2018], we assess generalization of SOTA AVR models including WReN [Barrett et al., 2018], CoPINet [Zhang et al., 2019b], RelBase [Spratley et al., 2020], SCL [Wu et al., 2020], MRNet [Benny et al., 2021], ALANS [Zhang et al., 2021], SRAN [Hu et al., 2021], PrAE [Zhang et al., 2022], CPCNet [Yang et al., 2023b], PredRNet [Yang et al., 2023a], STSN [Mondal et al., 2023], and DRNet [Zhao et al., 2024]. For direct comparison, we evaluate all models on I-RAVEN following the above-described experimental setup.

Reproducibility. To guarantee reproducibility of experiments, we use a fixed set of random seeds and turn off hardware and framework features concerning indeterministic computation wherever possible. Together with the code, we provide the full training script that can be used to run all training jobs. The training job is packaged as a Docker image with fixed dependencies to isolate the configuration of the training environment. The released code allows for generation of all datasets from scratch, eliminating the dependency on file-hosting services required to distribute the data. The code for reproducing all experiments is publicly accessible at: https://github.com/mikomel/raven

4.1 Generalization on A-I-RAVEN

Main regimes. In the first set of experiments we evaluate all considered models on 4 primary generalization regimes of A-I-RAVEN. The results are presented in Table 2, along with the reference results on I-RAVEN and I-RAVEN-Mesh. The best outcomes on A/Color and A/Position are achieved by DRNet, followed by RelBase and SCL that perform comparably. In A/Size and A/Type, SCL outperforms other models, with DRNet and RelBase taking the second and third place, resp. Interestingly, the top 3 models present a mix of architectures comprising large models, such as DRNet that includes a Vision Transformer backbone (24.7M params), as well as small models, such as SCL and RelBase that include mainly convolutional and feed-forward layers (0.6M

and 1.3M params, resp.). This suggests that various architectural approaches may be taken to achieve reasonable generalization performance in solving RPMs.

Extended regimes. Table 3 shows the aggregated performance of all considered models on 6 extended A-I-RAVEN regimes. Similarly to the main regimes, the best results are achieved by SCL, RelBase, and DRNet. Overall, replacing the Constant rule in the training set of the A/Color regime with Progression yields a dataset of slightly lower complexity, as the best model on A/Color-Progression achieved 75.6% accuracy, a 5.6 p.p. increase compared to the best result on A/Color. Conversely, using the Arithmetic and Distribute Three rules increases the difficulty, as measured by the drop of the max accuracy by 3.3 p.p. and 4.3 p.p., resp. Furthermore, using a pair of held-out attributes significantly increases the complexity. For instance, in A/ColorType, the most challenging regime, the best result is only 32.0%. We conclude that A-I-RAVEN provides a suite of challenging regimes of variable complexity, in which even the best-performing models are far from solving all test matrices.

Dataset difficulty. Across all A-I-RAVEN regimes, the highest average result was achieved by SCL (56.3%), followed by DRNet (54.8%) and RelBase (54.1%). While SCL achieved 83.4% test accuracy on I-RAVEN, on A-I-RAVEN regimes it scored from 32.0% on A/ColorType to 76.7% on A/Position. Similar differences can be observed for all remaining models, which shows that generalization regimes of A-I-RAVEN pose a bigger challenge than the base dataset.

Fig. 4 displays the difference in performance of top-3 models on test and validation splits. On I-RAVEN and I-RAVENMesh the difference is negligible, as in these datasets both splits follow the same distribution. However, the difference in attributeless regimes is significant, indicating the need for further research on generalization.

Tables 4 - 15 in [Małki´nski and Ma´ndziuk, 2025a, Appendix C] present the results of all considered models on test and validation splits and the difference between these two splits for particular datasets/regimes. The difference in model performance between test and validation splits in I-RAVEN (Table 4) and I-RAVEN-Mesh (Table 5) is negligible. In A-I-RAVEN regimes, however, the difference is significant, showing limitations of all evaluated models in terms of generalization. Across 4 primary regimes (Tables 6 - 9), the biggest difference concerns the A/Type regime, suggesting that generalization of rules applied to novel shape types constitutes a real challenge for the contemporary models. In all 3 extended regimes concerning held-out attribute pairs (A/ColorSize, A/ColorType, and A/SizeType) the performance difference on test and validation splits is bigger than in the primary regimes (see Tables 10 - 12). This drop stems from overall weaker performance on the test split, confirming high difficulty of these regimes. Model performance on the next 3 regimes concerning the Color attribute and rules other than Constant (A/Color-Progression, A/Color-Arithmetic, and A/Color-DistributeThree) is better, though further progress in generalization is required to fully close Accuracy ± std (%)

Figure 4: Dataset difficulty. Performance of top-3 models on test and validation splits. Numerical values refer to SCL scores.

the performance gap between test and validation splits (see Tables 13 - 15).

Per-configuration results. Tables 16 - 27 in [Małki´nski and Ma´ndziuk, 2025a, Appendix C] present the detailed results of all considered models for all matrix configurations. The most challenging configurations in I-RAVEN and IRAVEN-Mesh are 3x3Grid and Out-InGrid, in which image panels contain more objects than in the remaining configurations. Apparently, such setups require stronger reasoning capabilities to correctly identify the rules applied to multiple objects. Also, the results on the Left-Right and Up-Down configurations are relatively weaker in most regimes. In these configurations, rules may be applied to both matrix components (left/right and up/down, resp.), increasing the task complexity. This also concerns the Out-InGrid configuration in the A/Size regime, and the Out-InCenter configuration in the A/SizeType regime. Results in the A/Position regime are closeto-perfect in configurations comprising a single object in each component (Center, Left-Right, Up-Down, and Out-InCenter) and weaker in the remaining configurations (2x2Grid, 3x3Grid and Out-InGrid). This performance drop can be attributed to the fact that Position attribute can only be effectively applied to the 2x2Grid, 3x3Grid and Out-InGrid configurations allowing modification of the object's position. In the remaining configurations its application does not introduce any changes.

4.2 Progressive Knowledge Acquisition on I-RAVEN-Mesh

In the second set of experiments we employ I-RAVEN-Mesh to examine the TL ability of the best performing models. To this end, we consider variants of partial I-RAVEN-Mesh dataset with a fraction q ∈ { 1 64 , . . . , 1} of the training set and compare the performance of a model trained from scratch on a partial dataset to that of a model pre-trained on full IRAVEN and fine-tuned on the respective part of I-RAVENMesh. Fig. 5 shows that for q = 1 64 pre-training RelBase, MRNet, CPCNet, SRAN, and CoPINet on I-RAVEN leads to gains smaller than 15 p.p., whereas pre-training DRNet, SCL, PredRNet, and STSN improved their accuracy by 50.6, 34.8, 23.4 and 29.6 p.p., resp. In addition, TL clearly improved performance of DRNet, SCL, PredRNet, and STSN in all considered settings. In particular for q = 1 by 9.0, 5.9, 20.5, and 18.8 p.p., resp., indicating the models' capacity for knowledge reuse.

5 Conclusion

We investigate generalization capabilities of DL models in the AVR domain. To accelerate research in this area, we propose two RPM benchmarks. A-I-RAVEN introduces 10 generalization regimes of variable complexity that assess model's capability to solve matrices with rules applied to novel attributes at various levels of complexity (primary and extended regimes). Contrary to the existing PGM dataset, A-I-RAVEN features compositionality, offers a variety of figure configurations, and above all does not require substantial computational resources. I-RAVEN-Mesh overlays line-based patterns on top of the RPM, facilitating TL studies. Experiments on 13 strong literature AVR models reveal their limitations in terms of generalization. We believe that the introduced datasets complement existing RPM benchmarks and will foster progress in the AVR area.

Limitations and future work. In this work we study generalization and knowledge transfer in contemporary AVR models employing RPM datasets. While RPMs are by far the most popular AVR tasks, the AVR domain also includes other types of problems not covered in the paper [Małki´nski and Ma´ndziuk, 2023]. The Machine Number Sense dataset presents visual arithmetic problems [Zhang et al., 2020], VAEC defines an extrapolation challenge [Webb et al., 2020], while ARC proposes a set of diverse tasks in a few-shot learning setting [Chollet, 2019]. Similar studies could be performed on problems other than RPMs to test the performance and knowledge transfer abilities of AVR models in other problem settings.

Acknowledgments

This research was carried out with the support of the Laboratory of Bioinformatics and Computational Genomics and the High Performance Computing Center of the Faculty of Mathematics and Information Science Warsaw University of Technology. Mikołaj Małki´nski was funded by the Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme. This paper builds on the MSc thesis titled "Transfer learning in abstract visual reasoning domain" by Adam Kowalczyk from the Warsaw University of Technology, Warsaw, Poland.