Top-Down Guidance for Learning Object-Centric Representations

1 Introduction

Humans are skilled at decomposing visual scenes into the compositions of objects [Kahneman et al., 1992], which is crucial for humans' efficient understanding, predicting, and planning. Inspired by this property, Object-Centric Learning (OCL) seeks to achieve human-like representations in neural networks. Specifically, models are trained in a selfsupervised manner to represent visual signals, such as images or videos with a set of latent vectors which are referred to as

∗Corresponding author

'slots' [Greff et al., 2019; Locatello et al., 2020]. Ideally, each slot corresponds to an object in the scene.

Previous methods [Locatello et al., 2020; Jia et al., 2023; Singh et al., 2021; Jiang et al., 2023] have achieved considerable progress in OCL. However, we observe that the performance of existing methods is highly unstable across different scenarios. For example, in Figure 1, we present the cases where current models show sup-optimal object understanding and decompose scenes in an inferior manner: some objects are missed or split into parts. This expresses a concern that current models highly rely on the inductive biases of model structures and visual scenes to learn object-centric representations, posing challenges for adapting to various scenarios and tackling downstream tasks.

We attribute this issue to the fact that current OCL models typically adopt an auto-encoding paradigm that learns objectcentric representations by encoding images into slots and using these slots to reconstruct images. However, reconstruction loss does not tell apart objects: models do not need to decompose the scene according to objects. As a result, these models fail to learn distinguishable features at the backbone. For instance, in Figure 1 and 6, we visualize the output features of the backbone when the object-centric models are trained solely with reconstruction loss. It is observed that these features are blurry and fail to align with the edges of objects. Moreover, the features of small objects are particularly challenging to distinguish from the background. Such indistinguishable feature makes it difficult for the model to decide how to assign features to slots, resulting in suboptimal object-centric representations.

To address this issue, we propose to guide the model by an additional top-down pathway. This coincides with humans' perceptual learning process, where theories about human vision [Hochstein and Ahissar, 2002; Wolfe, 2021] argue that humans first learn concepts at high-level consciousness and then use the high-level perception to guide the learning of low-level neurons. Drawing on this mechanism, we propose Top-Down Guided Network (TDGNet) that introduces the top-down pathway to use the high-level representations (i.e., the slots) to guide the low-level features output by the backbone. Specifically, we obtain a high-level guidance by weighted summing the slots according to their masks, and then introduce a projection network to predict this guidance signal using low-level features. In this way, the model tends to cluster low-level features belonging to the same slots, and otherwise keep them apart, thus making low-level features more distinguishable. Moreover, we extend this concept to the inference phase, introducing a conflict detection method designed to refine object-centric representations during inference: when a feature's prediction is far from all existing slots or close to multiple slots, it may represent a suboptimal perception such as missing objects or splitting objects into parts, which we call a conflict. We detect and solve such conflicts by adding or merging slots, thus refining the object-centric representations.

We evaluate TDGNet and compare it with current SOTA models on multiple tasks. We first introduce CLEVRTex [Karazija et al., 2021], MOVi-C [Greff et al., 2022] and COCO to evaluate the object-centric representations, where TDGNet outperforms current SOTA models in terms of common object discovery metrics. Furthermore, we expand the downstream task scope of TDGNet by applying it to the field of robotics. We introduce RoboNet [Dasari et al., 2020] and VP2 [Tian et al., 2023], to evaluate TDGNet with downstream tasks including video prediction and visual planning, demonstrating that TDGNet adapts well to these tasks.

To sum up, our contributions are summarized as follows:

• Drawing on the top-down visual pathway of humans, we propose TDGNet, which incorporates a top-down pathway that constructs high-level guidance to optimize the low-level features, thereby improving object-centric representations.
• Based on the top-down pathway, we propose conflict detection to discover perceptual errors and further refine object-centric representations during inference.
• We demonstrate the SOTA performance of TDGNet in object-centric representation tasks. Besides, we introduce it into more complex robotic scenarios and verify

that it works effectively in the video prediction and visual planning task.

2 RelatedWork

Object-Centric Learning. Most current OCL methods follow an auto-encoding paradigm that first encodes input signals into several slots and reconstructs the original signal with these slots. Earlier works, including IODINE [Greff et al., 2019], MONet [Burgess et al., 2019] and GENESIS [Engelcke et al., 2019], accomplish this task by using multiple encoder-decoder structures. Slot-Attention [Locatello et al., 2020] proposed an iterative attention method that allows slots to compete for input image segments and conduct segmentation. A critical issue of current OCL methods is how to generalize to more complex scenes. BO-QSA [Jia et al., 2023], I-SA [Chang et al., 2022] and InvariantSA [Biza et al., 2023] focus on query optimization, which uses learnable parameters to initialize slots. SLATE [Singh et al., 2021] and LSD [Jiang et al., 2023] attempt to improve the decoder structure, introducing transformer-based and diffusion-based decoders to enhance the model's reconstruction ability. DINOSAUR [Seitzer et al., 2022] proposes that the simple reconstruction task is insufficient to distinguish objects and replaces the reconstruction objective with the output feature of DINO [Caron et al., 2021].

Top-down connections in human vision. Human brain transmits high-level semantic information to low-level neurons [Beck and Kastner, 2009], resulting in a biased competition between different objects to control attention. In addition, the brain can receive task-relevant information and inhibit irrelevant neurons to improve the efficiency of completing tasks [Karimi-Rouzbahani et al., 2017]. Reverse hierarchy theory [Hochstein and Ahissar, 2002; Ahissar and Hochstein, 2004] generalizes neural connections in the human brain into two functional pathways: the bottom-up path-

(b) Conflict detection (CD) for refining representations during inference.

Figure 2: Architecture of Top-Down Guided Network (TDGNet). The proposed TDGNet acquires the initial perception through the bottom-up pathway (black arrow) and refines its object-centric representations with the top-down pathway (red arrow). (a) During training, TDGNet uses slots and object masks to guide the backbone to produce distinguishable low-level features. (b) During Inference, TDGNet refines the slots by detecting conflicts between its slots and the low-level features.

way works implicitly, acquiring the gist of the scene rapidly, and the top-down pathway returns to lower-level neurons to bring detailed information into consciousness.

Top-down connections in Neural Networks. It has long been explored to incorporate top-down feedback connections into neural networks. [Liang and Hu, 2015] introduce recurrent connections into convolutional networks. [Wen et al., 2018] propose to achieve predictive coding through a network with feedback connections. Recent works have utilized a top-down pathway to solve different visual or multi-modal tasks, including semantic segmentation [Yin et al., 2022; Liu et al., 2024], visual saliency [Ramanishka et al., 2017] and vision question & answering [Anderson et al., 2018]. Most of these models integrate features from multiple layers through a parameterized network module, introducing additional depth into the network through feedback connections. However, there is no evidence that they achieve the visual functions of human feedback connections.

3 Method

3.1 Preliminary: Auto-encoding-based Object-Centric Models

The architecture of TDGNet is shown in Figure 2. A typical auto-encoding-based object-centric model serves as our bottom-up pathway: the backbone EB first extract the lowlevel features F ∈ RCf×H/s×W/s from the input image I ∈ R3×H×W . Then a slot encoder ES encodes F into K slots S ∈ RK×Cs , which is regarded as the high-level representations. Finally, a slot decoder DS decodes slots into reconstructions R ∈ R3×H×W and object masks M ∈ RK×H×W . Formally, the bottom-up pathway is described as:

The bottom-up pathway is optimized by reconstructing input images. Here we use a combination of L1 loss and perceptual loss (LPIPS) [Zhang et al., 2018] for optimization. The reconstruction loss is written as:

3.2 Learning Object-Centric Representations with Top-down Pathway

We introduce a top-down pathway to refine the object-centric representations, which works in two ways: during training, it constructs guidance with high-level slots to optimize lowlevel features output by the backbone; during inference, it refines the slots by detecting and solving conflicts between slots and low-level features.

Top-down Guidance during Training

We first introduce how the top-down pathway constructs the Top-Down Guidance (TDG) to optimize low-level features during training. As shown in Figure 2(a), the bottom-up pathway provides initial object-centric representations with slots. The top-down pathway utilizes these slots to construct guidance for training. Ideally, it makes features from the same slot more similar than those from different ones.

The guidance is constructed through high-level representations including object masks M and slots S. Formally, the high-level guidance FH is obtained through the sum of S, weighted byMat each spatial location:

Here SG represents the stop-gradient operation. We stop the gradient of high-level signals (namely M and S) so that the guidance only works on the low-level features.

Subsequently, we introduce a projection network P that uses low-level features to predict the high-level guidance FH. In this way, the low-level features are required to predict their corresponding slots. Considering that FH are produced with the slots S, it provides more accurate object regions for the reconstructed images R than the input images I. Therefore, we let P project the low-level features extracted from R for more accurate guidance. Formally, we re-input R into the backbone EB to extract its low-level features ̂F:

Finally, we use the projection network P to predict FH with ̂F. In our method, slots S are normalized to unit length, and the distance is measured through cosine similarity CosSim. The top-down guidance loss LTD is formulated as below:

Overall, TDGNet is trained by L, the weighted sum of the reconstruction loss and the top-down guidance loss:

Conflict Detection during Inference

During training, we require low-level features to predict their corresponding slots with the projection network P, which provides a method for the model to refine the slots during inference, which we call conflict detection (CD): Ideally, the prediction of a low-level feature should be close to its corresponding slot. This indicates two facts: (i) When a feature's prediction is far away from all the slots, it represents a wrong perception, such as an undiscovered object, and (ii) When a feature's prediction is close to multiple slots, these slots may represent a single object divided into multiple parts. We resolve such conflicts by adding and merging slots, thereby improving the object-centric representations.

Specifically, we first solve the first issue. After the model extracts low-level features F and slots S from the images with the bottom-up pathway, we use the projection network P(·) to produce a prediction P(F) and computes the cosine distance between S and P(F) and acquire the conflict C. Here we define C as the distance from P(F) to their nearest slot:

We set a threshold th to determine whether a conflict is large or not. Repetitively, we select ˆ f from F that has the largest conflict. If the conflict of ˆ f exceeds th, we add P( ˆ f) to the slot set. This process is repeated until all the conflicts are lower than th. After that, we use an agglomerative clustering algorithm to merge slots with less cosine distance than th, thus mitigating the situation where an object is divided into multiple parts. As for the choice of th, we propose a heuristic method that for each trained model, we calculate the average distance between slots and set th as half of this distance.

4 Experiments

In this section, we first evaluate the object-centric representations with the object discovery task, demonstrating TDGNet's superior object discovery performance across multiple scenes of varying complexity. We then evaluate the performance of the learned object-centric representations in downstream tasks such as unconditional/conditional video prediction and video prediction for visual planning. Specifically, for conditional video prediction and visual planning, we introduce robotics benchmarks [Dasari et al., 2020; Tian et al., 2023] to demonstrate that TDGNet's object-centric representations can be applied in a variety of scenarios.

4.1 Unsupervised Object Discovery

Setup. We first introduce the object discovery task for evaluating object-centric representations. We use three common datasets: CLEVRTex [Karazija et al., 2021], MOVi-C [Greff et al., 2022], and COCO [Caesar et al., 2018]. The model's generalization ability is also evaluated using CLEVRTexOOD and -CAMO, two out-of-distribution test sets. The complexity of the datasets varies. Objects in CLEVRTex have regular shapes, but the challenge is that it adds texture maps to objects and backgrounds, increasing the appearance complexity of objects. MOVi-C takes a further step to use realistic, richly textured objects from the GSO dataset [Downs et al., 2022] to create multi-object scenes. COCO contains a large number of natural scene images, and the variability of object appearance has increased significantly. Following previous works [Karazija et al., 2021], we use the foreground adjusted rand index (ARI-FG) [Rand, 1971], mean IoU (mIoU), and mean square error (MSE) to evaluate the models' ability to discover objects and reconstruct the images. For MOVi-C and COCO, we follow [Jiang et al., 2023; Seitzer et al., 2022] to include mean best overlapping (mBO) for evaluation.

Result. Table 1 displays quantitative comparison results on CLEVRTex and MOVi-C. In the comparison, we adopt two inference modes of TDGNet: one that only initiates the bottom-up pathway and another that applies the conflict detection (CD) operation to acquire a refined perception. TDGNet with only the bottom-up pathway, outperforms most current models. Conflict detection provides a further performance boost. TDGNet also generalizes well on CLEVRTexOOD and -CAMO, outperforming current SOTA models by a large margin.

On MOVi-C, SOTA models, such as LSD and DINOSAUR, rely on reconstructing features from pre-training backbones to generalize to complex scenarios. However, these models show significant biases in different evaluation metrics. For instance, DINOSAUR offers an ARI-FG far superior to other models but provides lower mIoU and mBO. On the other hand, the performance of LSD is just the opposite, with lower ARI-FG and higher mIoU and mBO. Our TDGNet, by contrast, outperforms previous models in all the metrics, demonstrating a more comprehensive object discovery capability. Figure 3 gives examples to illustrate the improvement. DINOSAUR fails to fit the edges and cannot segment the background holistically on MOVi-C. LSD tends to divide objects into multiple parts as is circled. TDGNet, instead, succeeds in segmenting backgrounds, as well as showing better object understanding where the object masks fit the outlines of objects well and do not divide objects into parts, thus achieving higher scores on all metrics.

We further provide the comparison result on the COCO dataset in Table 2. Following [Seitzer et al., 2022], we discuss the situations when different types of decoders are used. For the MLP-based decoder, we adopt DINOSAUR as the baseline and use DINO v2 as the backbone. Compared with DINOSAUR trained only through reconstruction, we have improved by 2.1, 0.7, and 1.2 respectively, according to ARIFG, mBO, and mIoU. For the transformer-based decoder, we choose SPOT as the baseline and combine it with TDGNet, achieving performance improvements of 0.3, 0.6, and 0.2 in terms of the three metrics.

4.2 Exploring Object-Centric Representation in Predicting the World's State

We believe that object-centric learning, which uses "objects" as the basic unit to represent visual scenes, is worth exploring in world modeling. Here we explore the field of robotics, introducing video prediction and visual planning tasks to evaluate whether the object-centric representations from TDGNet benefit the prediction of the world's states. In video prediction tasks, after training TDGNet, we additionally train an auto-regressive Transformer to predict the future slots, and then decode these slots to acquire future frames. Furthermore, the visual planning task evaluates the video prediction model by using the prediction results in control tasks.

Video Prediction

Setup. We conduct video prediction on the MOVi-C and RoboNet datasets. MOVi-C, a widely used multi-object dataset, contains videos of the interactions of moving objects. Following previous works [Song et al., 2024], the model predicts 8 future frames from 6 context frames with no conditions. We primarily compare TDGNet to existing ObjectCentric models in MOVi-C to ensure that the slots provided by TDGNet outperform existing models. RoboNet is a largescale robot dataset that contains a large number of videos of object manipulation with robotic arms. The models are required to predict 10 future frames given 2 context frames and the robotic arm's action as conditions. We compare TDGNet to existing conditional video prediction models on RoboNet. Following previous work, we use SSIM, PSNR, and LPIPS for evaluation on MOVi-C, and additionally introduce FVD for RoboNet.

Result. We list the comparison results in Table 3 and 4. For MOVi-C, using the slots extracted by TDGNet for prediction significantly improves the performance, outperforming other models by a large margin. For RoboNet, we provide results on 256 resolution. We observe that our model outperforms iVideoGPT, the current SOTAmodel, in three out of the four evaluation metrics. In Figure 4, we provide the visualization results on RoboNet, demonstrating that our model accurately simulates the interaction between robots and objects.

Visual Planning

Setup. As is discussed in previous works [Tian et al., 2023], the video prediction performance sometimes cannot reflect whether models correctly predict the world's state. Therefore, we use their proposed VP2 benchmark [Tian et al., 2023] to evaluate the performance of TDGNet in the visual planning task. In VP2 , a given agent uses the predicted frames from video prediction models to complete various control tasks. All elements are provided except the video prediction models, thus comparing them under a standardized benchmark. We use the RoboDesk environment in VP2 for evaluation, which includes 7 kinds of control tasks. The agent's success rate to complete the tasks is used as the evaluation metric.

Result. In Figure 5, we present a comparison between TDGNet and a set of baseline models [Wu et al., 2024; Babaeizadeh et al., 2021; Villegas et al., 2019; Voleti et al., 2022; Minderer et al., 2020; Gupta et al., 2022]. 'Simulator' is the success rate of the agent when it directly use the simulator as the dynamics, representing an upper bound of model performance. TDGNet attains performance that is comparable or superior to existing SOTA models across the majority of tasks. In addition, we compute the average success rate, normalized by the simulator's performance, demonstrating that TDGNet outperforms existing SOTA models.

4.3 Ablative Experiments

We conduct ablative experiments on the components of TDGNet during training and inference, including the L1 and LPIPS loss used for reconstruction, as well as the top-down guidance (TDG) and conflict detection (CD) performed by the top-down pathway during training and inference. According to the result in Table 5, the LPIPS loss improves the performance of baseline models trained only with L1 loss, particularly for MOVi-C, with improvements of 5.7, 10.4, and 11.8, respectively, in terms of ARI-FG, mBO, and mIoU. The top-down pathway further provides considerable performance gains. TDG and CD bring about improvements of 3.9, 2.4, and 2.9 on CLEVRTex, as well as 9.9, 10.3, and 10.7 on MOVi-C. The top-down pathway offers greater improvements on more complex multi-object scenes (i.e., MOVi-C), indicating that the top-down pathway in TDGNet tends to help the model overcome more challenging samples.

4.4 Analysis

Low-level Feature Optimization with TDG

We visualize the internal features of models in Figure 6(a) by taking the low-level features from TDGNet and the baseline model without TDG (i.e., BO-QSA [Jia et al., 2023]), using PCA [Abdi and Williams, 2010] to reduce the feature dimension to 3 for visualization. Feature maps from BO-QSA are fuzzy. Some of the small objects are almost unseen in the feature map. By contrast, the features extracted by TDGNet are more conducive to identifying individual objects. The features of the same objects are highly similar, while definite boundaries separate adjacent objects. Further in Figure 6(b), we calculate the intra- and inter-object feature variances, which confirms that TDGNet achieves a higher inter-object feature variance and a much lower intra-object feature variance, making objects more distinguishable in the low-level features.

Iterative Refinement with CD

Compared with extracting objects directly through attention competition, the process of detecting and resolving conflicts provides a more explainable way of object discovery. We visualize the process of CD in Figure 7. The model repeatedly computes conflicts between low-level features and slots and includes features with large conflicts into the slot set. To clearly illustrate this process, we manually corrupt the bottom-up pathway, assuming that no object is detected and only the background is given at first. In this extreme case, CD repeatedly calculates the conflicts and includes objects. After multiple iterations, all conflicts are solved, representing that all objects are included.

5 Conclusion

Observing that OCL models provide suboptimal objectcentric representations such as missing objects or splitting objects into parts, we analyze existing models and propose that OCL models trained solely with reconstruction loss cannot learn distinguishable low-level features. To address this issue, we refer to human vision and propose TDGNet, which introduces a top-down pathway that guides low-level features with the high-level representations (i.e., slots). We verify that the top-down pathway makes objects more distinguishable in the low-level features. Our experiment results show that TDGNet outperforms existing SOTA models in the object discovery task while also demonstrating its potential for downstream tasks such as prediction and planning in the robotics field.

Acknowledgments

This work was supported by Chinese National Natural Science Foundation Projects U23B2054, 62276254, 62206280, 62376265, the Beijing Science and Technology Plan Project Z231100005923033, Beijing Natural Science Foundation L221013, L242092, the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA0480103, the Youth Innovation Promotion Association CAS Y2021131 and InnoHK program.