Advancing Generalization Across a Variety of Abstract Visual Reasoning Tasks

1 Introduction

The abstract visual reasoning (AVR) domain encompasses visual tasks requiring reasoning about abstract patterns expressed through image-based analogies. A classical AVR task, Raven's Progressive Matrices (RPMs) [Raven, 1936; Raven and Court, 1998], illustrated in Fig. 1, consists of a 3×3 grid of panels with the bottom-right panel missing. Panels in the first two rows are designed according to some number of abstract rules that govern objects and attributes in the images. The task is to complete the grid by selecting the correct answer from the eight provided choices. Another AVR task, visual analogies, shown in Fig. 2, involves two rows of images. The top row presents an abstract relation that must be instantiated in the bottom row by selecting one of the four answer panels that correctly completes the analogy.

Solving AVR tasks involves detecting rule patterns across images, abstracting them into crisp concepts, and applying these concepts to novel scenarios. For example, matrices in the visual analogy problems (VAP) dataset [Hill et al., 2019] present different domains in matrix rows (e.g., shape type (top) and line type (bottom) in Fig. 2a), emphasizing the

(a) PGM [Barrett et al., 2018]

(b) I-RAVEN [Hu et al., 2021]

(a) VAP [Hill et al., 2019]

(b) VASR [Bitton et al., 2023]

Figure 2: Visual analogies.

importance of forming domain-independent concept representations. Such analogy-making abilities are closely tied to fluid intelligence [Snow et al., 1984; Carpenter et al., 1990; Lake et al., 2017], a cornerstone of human cognition. Replicating these capabilities in learning systems has been a long-standing goal of research in the field [Gentner, 1980; Hofstadter, 1995; French, 2002; Lovett et al., 2007; Gentner and Forbus, 2011].

A key aspect of AVR tasks, central to our work, are their systematic problem generation methods. Underneath each AVR task design lies a precise definition of its abstract structure, which defines the rule patterns expressed in the matrices.

For instance, each PGM matrix [Barrett et al., 2018] has a corresponding abstract structure S = {(r, o, a) | r ∈ R, o ∈ O, a ∈ A}, where R = {progression, XOR, OR, AND, consistent union}, O = {shape, line}, and A = {size, type, color, position, number} are the sets of rules, objects, and attributes, resp. This formal specification facilitates defining dataset splits with varying feature distributions, enabling the evaluation of generalization by training models on matrices with specific abstract structures and testing them on matrices with different structures.

Motivation. Several AVR studies have addressed the i.i.d. problem formulation, where models are trained and tested on matrices sampled from a shared feature distribution. Continuous improvements have produced methods that surpass human performance on tasks like RPMs, given sufficient amount of training data [Hern´andez-Orallo et al., 2016; Małki´nski and Ma´ndziuk, 2025c]. Other research lines have demonstrated the benefits of knowledge transfer [Ma´ndziuk and ˙Zychowski, 2019; Tomaszewska et al., 2022] and multitask learning [Małki´nski and Ma´ndziuk, 2024b]. Despite these achievements, o.o.d. problem formulations, where models are evaluated on matrices sampled from a different feature distribution than the one used for training, remain a major challenge even for state-of-the-art (SOTA) deep learning (DL) models. Moreover, existing approaches in the AVR domain primarily target synthetic tasks with simple 2D geometric shapes, without considering their applicability to problems with real-world data. In this work, we strive to develop a model architecture that not only performs well in i.i.d. tasks but also excels in o.o.d. settings. Additionally, we consider both synthetic and real-world setups to broaden the applicability of the proposed approach.

Contribution. To tackle these open challenges, we introduce the following contributions:

• We propose Pathways of Normalized Group Convolution (PoNG), a new neural model for AVR tasks that integrates group convolution, normalization, and a parallel design.
• We perform a comprehensive evaluation of PoNG against a wide range of SOTA models. The experiments show PoNG's versatility in both i.i.d. and o.o.d. problem setups, spanning RPMs and VAPs in synthetic and real-world scenarios.
• We conduct an ablation study to analyze the contributions of PoNG's modules, providing deeper insights into its design.

2 RelatedWork

AVR tasks. The AVR domain comprises a wide range of challenges [Mitchell, 2021; van der Maas et al., 2021; Stabinger et al., 2021; Małki´nski and Ma´ndziuk, 2023]. Most relevant to our work are tasks involving RPMs and visual analogies. After the introduction of early RPM datasets [Matzen et al., 2010; Wang and Su, 2015; Hoshen and Werman, 2017], two large-scale benchmarks were developed and broadly adopted in the DL literature. PGM [Barrett et al., 2018] (Fig. 1a) introduced 8 regimes to measure generalization of DL models. RAVEN [Zhang et al., 2019a] presented matrices with hierarchical structures across 7 figure configurations. Subsequent works further expanded the RAVEN dataset line: I-RAVEN [Hu et al., 2021] (Fig. 1b) mitigated a bias in RAVEN's answer generation method, AI-RAVEN [Małki´nski and Ma´ndziuk, 2025a] (Fig. 3) defined 10 generalization regimes of varying complexity, and I-RAVEN-Mesh [Małki´nski and Ma´ndziuk, 2025a] (Fig. 4) overlayed line-based patterns on top of the matrices. A parallel research stream introduced the VAP dataset [Hill et al., 2019] (Fig. 2a), which similarly to PGM enables measuring generalization on matrices with a different structure. VASR [Bitton et al., 2023] (Fig. 2b) introduced analogies with real-world images requiring understanding of rich visual scenes. A detailed description of the datasets used in this work is provided in Section 4.1.

AVR solvers. Early attempts to solve AVR tasks with DL models were prompted by the development of Relation Network (RN) [Santoro et al., 2017]. WReN [Barrett et al., 2018] applies RN to panel embeddings, CoPINet [Zhang et al., 2019b] integrates RN with contrastive mechanisms, and Conv1D Conv1D

2×

2×

ReLU

BN1D

G-C

GP-C

ReLU

BN1D

ReLU

BN1D

(b) Pathways block

(a) End-to-end PoNG architecture

Conv1D

. . .

Conv1D

TCN

. . .

TCN

(c) Group convolution (P3)

(d) Group-pair convolution (P4)

Figure 5: PoNG. (a) The panel encoder embeds each input image xi independently, producing hi. Context panel embeddings {hi}8i=1 together with the embedding of k'th answer hk are stacked and processed with the reasoner, leading to zk. (b) The pathways block, a key component of PoNG, comprises four parallel pathways P1 - P4. (c) P3 and (d) P4 employ novel normalized group convolution operators. PosEmb denotes position embedding, G-C the group convolution module used in P3, and GP-C the group-pair convolution module used in P4. The red dashed line marks the point after which G-C and GP-C perform analogous computation.

MRNet [Benny et al., 2021] embeds RN into a multi-scale architecture. Differently, SRAN [Hu et al., 2021] processes distinct panel groups with dedicated ResNet encoders, SCL [Wu et al., 2020] splits embeddings into groups processed by a shared neural layer, RelBase [Spratley et al., 2020] primarily relies on convolutional layers, ARII [Zhang et al., 2022b] learns robust rule representations via internal inferences, CPCNet [Yang et al., 2023b] utilizes a self-contrasting learning process to align perceptual and conceptual input representations, PredRNet [Yang et al., 2023a] mimics the prediction and matching process, and DRNet [Zhao et al., 2024] merges panel representations of two independent encoders. Other works develop neuro-symbolic approaches [Zhang et al., 2021; Zhang et al., 2022a] or perform explicit object recognition prior to reasoning [Mondal et al., 2023; Mondal et al., 2024]. Neural Structure Mapping [Shekhar and Taylor, 2022] decouples perception from reasoning to solve visual analogies. Bitton et al. [2023] formulate several zeroshot and supervised methods to solve real-world analogies using a frozen pre-trained Vision Transformer [Dosovitskiy et al., 2021] as the panel encoder. Although diverse approaches have been tried to tackle AVR benchmarks, contemporary models continue to exhibit limitations in generalization. In this context, we propose PoNG, a new versatile AVR model that performs well across diverse tasks.

3 Pathways of Normalized Group Convolution (PoNG) Model

We introduce PoNG (Fig. 5), a novel model that outcompetes baselines across a number of problem settings. The model follows a typical two-stage design. Firstly, it generates an embedding of each image panel. Then, it aggregates representations of matrix panels to predict the index of the correct answer. The details are described in [Małki´nski and Ma´ndziuk, 2025b, Appendix A].

Let (X, y, r) denote an AVR matrix, where X = {xi}ni=1 is the set of n image panels comprising nc context panels {xi}nc i=1 and na answer panels {xi}ni=nc+1 , xi ∈ [0, 1]h×w , i = 1, . . . , n is a grayscale image of height h and width w, y ∈ {0, 1}na is the one-hot encoded index of the correct answer, r ∈ {0, 1}dr is the multi-hot encoded representation of matrix rules of dimensionality dr using sparse encoding [Małki´nski and Ma´ndziuk, 2024a]. For RPMs nc = na = 8, for VAP nc = 5, na = 4, and for VASR nc = 3, na = 4. In each experiment h = w = 80, while dr is determined by the number of different abstract structures in the corresponding dataset (dr = 40 for I-RAVEN and AI-RAVEN, dr = 48 for I-RAVEN-Mesh, dr = 50 for PGM, and dr = 28 for VAP).

Panel encoder. The first component of the model has the form E : x → h, where h ∈ Rdh is the input panel embedding of dimensionality dh. Following RelBase [Spratley et al., 2020], the module comprises 2 blocks of the same architecture. Each block includes 2 parallel pathways that build high-level and low-level features, resp. The first one contains 2 convolutional blocks, each with 2D convolution, ReLU, and Batch Normalization (BN) [Ioffe and Szegedy, 2015]. The second one contains 2D max pooling followed by 2D convolution. The sum of both pathway results forms the block output. Differently from RelBase, we flatten the height and width dimensions of the resultant embedding, pass it through a linear layer with ReLU, flatten the channel and spatial di-

2×

+

+

mensions, and pass the tensor through a feed-forward residual block with Layer Normalization (LN) [Ba et al., 2016]. Finally, we concatenate the tensor with a position embedding (a learned 25-dimensional vector for each panel in the context grid), leading to h.

Reasoner. The second component of the model has the form R : {hi}8i=1 ∪ hk → zk, where hk is the panel embedding of k'th answer. For each answer panel, the reasoner produces embedding zk that describes how well the considered answer fits into the matrix context. Panel embeddings {hi}8i=1 ∪ hk are stacked and processed by a sequence of 3 reasoning blocks interleaved with 2 bottleneck layers for dimensionality reduction. Each reasoning block comprises BN and 4 parallel pathways, outputs of which are added together to form the output of the block. Next, the latent representation is passed through adaptive average pooling, flattened, processed with a linear layer with ReLU, passed through BN and projected with a linear layer to zk ∈ R128 .

Pathways. The key aspect of the reasoner module are its pathways. Each takes an input tensor of shape (B,C,D), where B is the batch size, C is the number of channels, and D is the feature dimension. In the first reasoning block D = dh and C corresponds to the number of panel embeddings in the considered group (C = 9 for RPMs, C = 6 for VAP, and C = 4 for VASR). Pathways are described as follows: P1 - a pointwise 1D convolution layer that mixes panel features at each spatial location; P2 - a sequence of 2 blocks, each comprising 1D convolution, ReLU, and BN, that builds higher level features spanning neighbouring spatial locations; P3 - analogous to P2, but 1D convolution is replaced with a group 1D convolution that splits the tensor into several groups along the channel dimension, applies a 1D convolution with shared weights to each group, and adds together the representations of each group; P4 - analogous to P3, but groups are arranged into pairs concatenated along the channel dimension and processedwith a 1Dconvolutionwith sharedweights. In contrast to [Krizhevsky et al., 2012], the proposed group convolution layers in both P3 and P4 apply TCN [Webb et al., 2020] to the outputs in each group. In the first layer P3 and P4 split the input tensor into 3 groups for RPMs and visual analogies, and into 2 groups for VASR, which allows for producing embeddings of each matrix row and each pair of rows, resp. Though we apply the pathways block in the AVR context, we envisage it as a generic module, also applicable to other settings involving a set of vector representations of shape (B, C, D).

Answer prediction. Representations of the context matrix filled-in with the respective answer, {zk}na k=1 , are processed with three prediction heads. The target head P y : zk → ̂ yk employs two linear layers interleaved with ReLU to produce score ̂ yk ∈ R describing how well the answer k aligns with the matrix context. The aggregate rule head P r 1 : {zk}na k=1 → ̂ r1 computes the sum of inputs and processes it with two linear layers interleaved with ReLU, producing a latent prediction of matrix rules ̂ r1 ∈ Rdr . We also introduce a novel targetconditioned rule head P r 2 : {zk}na k=1 → ̂ r2, which processes its input through a linear layer and computes a weighted sum of the resultant embeddings with weights given by the predicted probability distribution over the set of possible answers σ({ ̂ yk}na k=1 ), where σ denotes softmax. The model is trained with a joint loss function L = CE(σ({ ̂ yk}na k=1 ), y) + βBCE(ζ( ̂ r1, r))+γBCE(ζ( ̂ r2, r)), where ζ denotes sigmoid, CE cross-entropy, BCE binary cross-entropy, β = 25 and γ = 5 are balancing coefficients.

4 Experiments

We employ a set of diverse AVR tasks to evaluate PoNG's generalization capabilities. Section 4.1 introduces the selected datasets, Section 4.2 details the experimental setup, and Section 4.3 presents the results.

4.1 AVRDatasets

AVR models are typically evaluated on RPM benchmarks, a problem set well-established in the literature. We utilize four RPM datasets: PGM [Barrett et al., 2018], I-RAVEN [Hu et al., 2021], I-RAVEN-Mesh and A-I-RAVEN [Małki´nski and Ma´ndziuk, 2025a]. We extend the evaluation of PoNG beyond RPMs, to two benchmarks comprising visual analogies with both synthetic [Hill et al., 2019] and real-world [Bitton et al., 2023] images.

PGM. The PGM dataset was the first large-scale RPM benchmark designed to evaluate the AVR capabilities of deep learning models. In PGM each matrix is defined by an abstract structure encompassing its rules, objects, and attributes. To assess generalization, the dataset is divided into 8 generalization regimes. In the Neutral regime, the train, validation, and test splits share the same feature distribution, constituting an i.i.d. learning challenge. In the remaining regimes, the train and validation splits share a common distribution, while the test split relies on a different distribution, enabling the evaluation of generalization to unseen feature combinations. Each regime contains 1.42M RPMs, where 1.2M, 20K, and 200K belong to the train, validation, and test splits, resp.

I-RAVEN. The RAVEN dataset [Zhang et al., 2019a] was constructed to expand the range of visual configurations in RPMs. It incorporates 7 configurations that define object locations within the matrices. For instance, in the Left-Right configuration, each panel is divided into left and right parts that can be governed by distinct rules. A subsequent study identified a bias in the RAVEN's answer generation method, enabling models to learn shortcut solutions [Hu et al., 2021]. To alleviate this issue, the I-RAVEN dataset was proposed, which employs an impartial answer generation method. We utilize I-RAVEN in the experiments to avoid learning shortcut solutions. The benchmark consists of 10K matrices per configuration, totaling 70K matrices, split into the train, validation, and test with a 60/20/20 ratio.

I-RAVEN-Mesh. The I-RAVEN-Mesh dataset builds upon I-RAVEN by rendering a grid of 1 to 12 lines on the underlying matrices. The grid is defined by two attributes, the number of lines and their position. While the dataset was originally introduced to assess knowledge acquisition in transfer learning settings, we use it for standard supervised learning. In this setup, the model is trained directly on the dataset, analogously to I-RAVEN, to expand the scope of the considered i.i.d. tasks.

A-I-RAVEN. The A-I-RAVEN dataset was introduced to combine the generalization assessment capabilities of PGM with the broad adoption of RAVEN-like benchmarks. Drawing from PGM, A-I-RAVEN defines 10 generalization regimes. In each regime, a subset of attributes follows specific rules in the train and validation splits, while being governed by different rules in the test split. For example, in the A/ColorSize regime, the Color and Size attributes adhere to the Constant rule in the train and validation splits and are governed by a rule other than Constant in the test split. This approach enables the evaluation of models on RPMs with novel rule-attribute combinations that were not seen during training. Each regime contains 70K matrices, analogously to I-RAVEN.

VAP. The VAP benchmark was introduced to assess the analogy-making capabilities of learning systems. Each VAP matrix consists of a 2 × 3 grid of panels. The task is to identify a concept in the source domain (top row) and instantiate it in the target domain (bottom row) by selecting the correct answer panel to complete the matrix. The dataset defines five generalization regimes: Novel Domain Transfer, Novel Target Domain: Colour of Shapes, Novel Target Domain: Type of Lines, Novel Attribute Values: Interpolation, and Novel Attribute Values: Extrapolation, which test the model's generalization to novel domains or attribute values. Each regime contains 710K matrices, with 600K, 10K, and 100K devoted to the train, validation, and test splits, resp. In all experiments we use the learning analogies by contrasting (LABC) dataset variant, which constructs the answer set using semantically plausible images that consistently complete the target domain with some relation.

VASR. The VASR dataset features visual analogies involving real-world images, requiring the learner to understand complex real-world scenes before solving the analogy problem. Each matrix consists of a 2 × 2 panel grid, with the bottom-right image missing. The task is to complete the matrix by selecting the correct image from the 4 provided choices. VASR follows the classical analogy problem formulation, which aims to complete the following relation: A is to B, as C is to D. We use Silver data for training, which includes

150K, 2.25K, and 2.55Kmatrices in the train, validation, and test splits, resp. Experiments are conducted on both dataset variants, featuring random and difficult distractors, resp.

4.2 Experimental Setting

We assess PoNG's generalization by comparing its performance to SOTA models on the respective datasets. PoNG is trained using a standard training strategy involving the Adam optimizer [Kingma and Ba, 2014] with default hyperparameters (λ = 0.001, β1 = 0.9, β2 = 0.999, ϵ = 10-8 ). Learning rate λ is reduced by a factor of 10 after 5 epochs without improvement in validation loss. Early stopping is applied after 10 epochs without validation loss reduction. We use batch size B = 128 for experiments on RAVEN-like datasets and B = 256 in the remaining cases to reduce training time on large datasets. All experiments are performed on a single GPU (NVIDIA DGX A100).

To ensure reproducibility, we use a set of fixed random seeds, provide a list of commands for running training jobs, and explicitly list static dependencies in configuration files. The code for reproducing all experiments is publicly accessible at: https://github.com/mikomel/raven

4.3 Results

Results on I-RAVEN and I-RAVEN-Mesh. We begin with evaluating PoNG in the i.i.d. setting on the I-RAVEN and I-RAVEN-Mesh datasets, comparing it to 13 SOTA baselines. As presented in Table 1, on I-RAVEN, using our experimental setup, PoNG achieves a test accuracy of 95.9%, outperforming all other models. When compared to results obtained with model-specific experimental setups (I-RAVEN† ), PoNG is placed just behind CPCNet, DRNet, and PredRNet, which achieve slightly better scores. PoNG also secures the 1st place on I-RAVEN-Mesh, demonstrating high capacity to handle matrices with rules that span a large number of objects. Unlike many baseline models that rely on deeper architectures such as DRNet, SRAN or STSN, PoNG presents competitive performance despite its parameter-efficient design. These results demonstrate PoNG's strong ability to solve i.i.d. RPM tasks.

Results onA-I-RAVEN. To assess generalization, we evaluate PoNG on the 4 primary regimes of the A-I-RAVEN dataset, where the training and test distributions differ significantly. As shown in Table 1, PoNG outperforms all baselines across all settings, achieving test accuracies ranging from 59.4% on A/Type to 80.3% on A/Color, surpassing the best reference models by 10.3 and 9.9 p.p., resp. Additionally, Table 2 shows PoNG's performance across 6 extended regimes, which cover more challenging generalization tasks. Similarly, PoNG achieves superior performance in all but one regimes. Notably, PoNG outperforms the 2nd best model in the A/Color-D3 regime by 15.6 p.p. Overall, the results on A-I-RAVEN highlight PoNG's ability to perform well across a wide range of generalization tasks with varying levels of complexity. However, certain regimes such as the 3 extended regimes with held-out attribute pairs (A/ColorSize, A/ColorType, A/SizeType) continue to pose a significant challenge for all models (including PoNG), raising the need for further advances in generalization.

Results on PGM. Table 3 presents PoNG's results across PGM regimes. The model achieves strong results in several settings, particularly excelling in the Held-out Triple Pairs regime, where it surpasses the best reference model by 19.4 p.p. On average, PoNG scored 57.3% accuracy securing the 2nd place, just behind DRNet with 58.3%. These results confirm PoNG's ability to perform well on RPM-based generalization challenges extending beyond the RAVEN dataset line. Synthetic visual analogies. Table 4 presents PoNG's results across 5 regimes from the VAP benchmark. PoNG achieves SOTA results in 3 out of 5 settings when compared to PredRNet, the currently leading VAP model. The Novel Attribute Values: Extrapolation regime poses the greatest challenge among VAP regimes, aligning with findings from PGM, where Extrapolation is also one of the most demanding regimes. Overall, PoNG and PredRNet perform competitively, with PredRNet achieving a better average score by 1 p.p. PoNG's strong results on VAP highlight its versatility in generalization tasks that extend beyond RPMs.

Real-World visual analogies. To evaluate PoNG on the VASR dataset we followed the approach proposed by the VASR authors and employed the Vision Transformer (ViT) [Dosovitskiy et al., 2021] as a perception backbone that produces image embeddings. Specifically, we used the same model variant as [Bitton et al., 2023], which is ViT-L/32 pre-trained on ImageNet-21k at resolution 224x224 and finetuned on ImageNet-1k at resolution 384x384. We replaced the panel encoder of PoNG with this frozen pre-trained backbone and trained the rest of the model from scratch. The results are presented in Table 5. The three reference methods perform comparably to each other, with Supervised Concat being slightly inferior to Zero-Shot methods on the random distractor split and slightly superior on the difficult split. However, in both dataset variants PoNG significantly outcompetes the strongest reference result with 92.0% vs. 86.0% and 70.5% vs. 54.9%, resp. This suggests that the proposed reasoner block is much more effective in reasoning over pretrained embeddings than baseline methods. The results support the claim that PoNG is a versatile model with strong analogical reasoning capabilities, applicable to both synthetic and real-world domains.

Ablation study. We performed an ablation study on the RAVEN dataset line to evaluate the contributions of different PoNG components. Table 6 summarizes the results. The removal of P1 and P2 (cf. Fig. 5) leads to performance drop, in particular on I-RAVEN-Mesh (-14.9 p.p.) and A/Size (-15.2 p.p.). Similarly, removing P3 and P4 reduces model performance, especially on A/Type (-5.5 p.p.). Disabling TCN leads to generally worse results, primarily on A/Color (-4.9 p.p.) and A/Size (-6.9 p.p.). As shown in [Małki ´nski and Ma´ndziuk, 2025b, Appendix B], PoNG w/o TCN may fail to generalize rules to held-out attributes. Training without P r 1 (β = 0) or P r 2 (γ = 0) typically reduces model performance, but training with one of these rule-based prediction heads compensates to some degree the lack of the other. However, the removal of both (γ = 0 ∧ β = 0) deteriorates results across all datasets, signifying high relevance of the auxiliary training signal in PoNG's training. Overall, the ablation study demonstrates that all employed design choices contribute to the model performance.

5 Conclusion

Generalization to novel problem types is an active and open area of DL research. In this work, we introduced PoNG, a novel AVR model that leverages group convolution, parallel design, weight sharing, and normalization. To evaluate its effectiveness and versatility, we conducted experiments on four RPM benchmarks and two visual analogy datasets comprising both synthetic and real-world images. PoNG demonstrates strong performance across all considered problems, often surpassing the state-of-the-art reference methods.

Future directions. We believe that the proposed pathways block, a key component of PoNG, is a generic module also applicable to other tasks that require reasoning over a set of objects (vector embeddings). Nevertheless, the presented experimental evaluation of PoNG is focused on variable RPM benchmarks, including I-RAVEN, I-RAVEN-Mesh, AI-RAVEN, and PGM, and two visual analogy datasets, i.e. VAP and VASR. Assessing the model's performance on problems outside the AVR domain constitutes an interesting continuation of this work.

Acknowledgments

This research was carried out with the support of the Laboratory of Bioinformatics and Computational Genomics and the High Performance Computing Center of the Faculty of Mathematics and Information Science Warsaw University of Technology. Mikołaj Małki´nski was funded by the Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme.