CROP: Towards Distributional-Shift Robust Reinforcement Learning Using Compact Reshaped Observation Processing

1 Introduction

To safely deploy machine learning (ML) methods in realworld scenarios, generalization is an important challenge. As training data cannot contain all possible situations in general, ML methods should be able to generalize to unseen samples instead of overfitting to the training data. More specifically, their learned behavior should be robust to scenarios not included in the training data, often also refereed to as out-of-distribution (OOD) generalization [Hendrycks et al., 2021; Quinonero-Candela et al., 2008]. Whilst important for supervised- and unsupervised-learning tasks, said generalization is especially important for the safe application of reinforcement learning (RL), where unexpected observations may cause unintended and potentially unsafe behavior [Amodei et al., 2016]. For instance, a shift might be present when transferring a robotics model from the training simulation to the real-world [Zhao et al., 2020]. Generally, increased generalization from a few examples, also referred to as few-shot learning, is induced by altering the training data to increase its diversity and impede overfitting [Wang et al., 2020]. Those data augmentation techniques have been successfully applied to both supervised and reinforcement learning by increasing the information used for training the model [Laskin et al., 2020]. However, while the enlargement of the data may prevent overfitting and thus improves robustness to OOD samples, the increased complexity also increases training difficulty and hinders optimal convergence. On the contrary, despite tracing insufficient generalization to overfitting to the training data, occurrences of benign overfitting have been shown and characterized [Bartlett et al., 2020]. Also, the states observed by the agent might not resemble an optimal representation, an "issues that are so often critical to successful applications" of reinforcement learning [Sutton and Barto, 2018].

Extending on these insights, we propose to apply Compact Reshaped Observation Processing (CROP) to reduce the observation such that presumably irrelevant details are removed and the remaining information is sufficient for learning a robust policy that generalizes to unseen shifted observations. Overall, we provide the following contribution:

• We formulate three concrete CROP methods applicable to fully observable state and action spaces, reducing the information with regards to the agent's position within the environment, the agent's action space and affect, and the objects within the environment (Figure 1).
• We provide proof-of-concept and show the strengths of each CROP over the full observation in a safety gridworld training environment and a shifted test environment.
• We evaluate the proposed CROPs and compare their effect in zero-shot generalization to the state-of the art of data augmentation in randomly generated mazes.

2 Background

2.1 Distributional Shifts

One key assumption in ML is that the data for training, testing and runtime is independent and identically distributed

[Bishop and Nasrabadi, 2006]. Then, good performance in individual stages of the process would imply a similar performance in all other stages [Malinin et al., 2021]. In practice, however, samples encountered during runtime may be out-of-distribution, e.g. due to sensors degrading over time [Hendrycks et al., 2021]. Distributional Shifts describe a problem where all samples are out-of-distribution, i.e. the whole distribution shifts during runtime [Quinonero-Candela et al., 2008]. This shift might be a slight unnoticeable change, or a significant alteration. Therefore, the key assumption of identically distributed data does not hold and performance may be impacted negatively. In safety-critical environments, this can cause severe issues, especially if the ML model makes unintended mistakes due to unexpected changes [Leike et al., 2017]. This paper focuses on solutions referred to as zero-shot generalization, where, given limited amount of training data, generally applicable solutions for conceptually similar problems shall be inferred.

2.2 ProblemFormulation

Fully observable decision making problems can be formulated as Markov Decision Process (MDP) MMDP = ⟨S,A, T ,R⟩ , where S is a set of states st from a feature space F, A is a set of actions at, T is the transition function T : S × A ↦→ ∆(S) and R the reward function R : S ×A ↦→ R [Puterman, 1990]. Additionally, we consider a shifted set of states S + ̸= S that may be encountered alternatively. The goal is to find a policy π : S ↦→ ∆(A), which maximizes the value V π (st) = Eπ[Gt|st] for all st ∈ S, where Gt = ∑∞t=0 γ tR(st, at) is the return and γ = [0, 1] is the discount factor. An optimal policy π∗ has the optimal value function V π∗ = V ∗ satisfying V ∗ ≥ V π′ for all st ∈ S and π′ .

MDPs can be extended to Partially Observable Markov Decision Processes (POMDP) MPOMDP = ⟨S,A, T ,R,Ω,O, b0⟩, additionally consisting of a set Ω of observations ot, observation function O : S ×A ↦→ ∆(Ω), and initial state distribution b0 → ∆(S), where the agent does not perceive the true state st of the environment, but only a noisy observation ot ∈ O according to O(ot|st, at-1) [Kaelbling et al., 1998]. Note that the augmentation function CROP defined later resembles O in its partiality.

2.3 Reinforcement Learning

Reinforcement learning (RL) is an experience-based approach to find optimal policies π∗ using experience tuples et = ⟨st, at, R(st, at), st+1⟩.

Actor-critic methods are common RL algorithms using a function approximator ˆπθ ≈ π∗ with learnable parameters θ, which are updated via gradient ascent according to gradient g [Sutton et al., 2000]:

where ˆ A(st, at) = Qˆπθ (st, at) - V ˆπθ (st) is the advantage and Qˆπθ is the action value function. In practice, the return Gt is used to approximate Qˆπθ (st, at) [Mnih et al., 2016].

Proximal policy optimization (PPO) is a modified version of the standard Advantage Actor-Critic (A2C) method according to (1), which iteratively minimizes a surrogate loss function LPPO to ensure stable learning [Schulman et al., 2017]:

where rt = ˆπθ(at|st) ˆπθ,old(at|st) is the importance sampling ratio of the current and old action probability with ˆπθ,old being the policy originally collecting experience samples et for the update, and ϵ < 1 is a clipping parameter, ensuring bounded changes θ to mitigate divergence.

3 RelatedWork

3.1 Robustness and Generalization in RL

Training robust RL agents that act reliably in unknown situations is a known challenge. One branch of research deals with the recognition of distributional shift [Ramanan et al., 2021] and unknown states [Pimentel et al., 2014; Thulasidasan et al., 2021]. If distributional shift or unknown states are detected, one solution is to have the RL agent ask a (human) supervisor for advice [Najar and Chetouani, 2021] or to adjust the RL agent, e.g., through further training. In Adversarial RL [Pinto et al., 2017], an opponent policy partially controls the agent during training, with the aim of minimizing the long-term reward. The result is a policy that is more robust to changes in the environment, as it anticipates intervention. Our approach does not detect distributional shift or include adversaries. Instead, it aims to increase robustness via zero-shot generalization.

Another branch of research aims to improve generalization by avoiding overfitting to the training data. Here, a variety of methods has been developed: stopping the training early [Raskutti et al., 2014], dropping random parts of the underlying neural network [Srivastava et al., 2014], or augmenting the training data with noise [Karystinos and Pados, 2000]. Furthermore, training can be carried out in as many different environments as possible [Tobin et al., 2017; Cobbe et al., 2020; Gissl´en et al., 2021], such that an RL agent can not succeed by overfitting to a small number of trajectories and is forced to acquire transferable knowledge. However, this requires a huge number of different environments, that are typically created by exhaustively generating variations of the same procedural environment. This is not sample efficient and makes it difficult to create distinct tests. However, the lack of distinct tests both diminishes the impact of distributional shift and the need for strong generalization to succeed. Thus, we aim to improve generalization by training with a reduced set of more relevant training data.

3.2 Data Augmentation in RL

Quantity and quality of the dataset heavily impact the training in ML [Ying, 2019]. Data Augmentation can be used to artificially increase the diversity in training data when only a limited amount of data is available. The idea is to systematically modify the training data to avoid homogeneous structures [Shorten and Khoshgoftaar, 2019]. In image recognition tasks, this is done by geometric transformations such as mirroring, rotating or hiding pixels. This serves as a regularization against overfitting and increases data efficiency, ultimately improving generalization [Yarats et al., 2021; Laskin et al., 2020].

Various approaches adopted these ideas to RL [Raileanu et al., 2021; Chen et al., 2020; Yarats et al., 2022]. Similar to our approach, Reinforcement Learning with Augmented Data (RAD) [Laskin et al., 2020] augments observations without domain specific knowledge or changes to the RL algorithms. In addition to image based observations, RAD proposes two methods for state based observations: random amplitude scaling and Gaussian noise. By creating more diverse training data, the authors report increased data efficiency and better generalization to unseen environments.

We propose three additional methods to augment state based observations that reduce the amount of distinct training observations instead of increasing it. Improvements of the training are a welcome positive effect, but we focus on zero-shot generalization to unseen environments.

3.3 Partial Observability and Invariants

In real-world problems, RL agents often face incomplete and imperfect information [Choi et al., 2019] and thus may perceive different states as similar [Spaan, 2012]. In such POMDPs, learning optimal policies with naive approaches is difficult, and the respective stochasticity is regarded as a fundamental challenge for RL [Vlassis et al., 2012; Ghosh et al., 2021; Jaakkola et al., 1994]. However, we propose to train with limited state information on purpose, as this potentially mitigates the effect of directly perceiving the distributional shift and thus may improve robustness. Naturally, the remaining state information must be sufficient to find an optimal policy.

Removing potentially irrelevant information to improve the training is common in other ML areas such as facial recognition [Chen et al., 2014]. While less common in RL, recent approaches followed this concept and proposed to explicitly learn invariants. [Zhang et al., 2021] showed that agents can learn observation representations in latent space which encode task-relevant information. [Agarwal et al., 2021] use a policy similarity metric (states are similar if the optimal policy has a similar behavior in that and future states) with a contrastive learning approach to learn policies invariant to observation variations. [Mazoure et al., 2022] use clustering methods and self-supervised learning to define an auxiliary task, which is mapping behaviorally similar states to similar representations. In fact, all these approaches require different observations from multiple training contexts and a complex nonlinear encoder that maps observations to a latent representation. On the contrary, we rely on a less complex handcrafted reduction of state information.

4 CROP

To facilitate generalization to mechanics underlying the environment, we propose to reshape observations to a compact format containing information with specific relevance to the agent. We argue, that if the reshaped state is invariant in similar situations, the policy optimization benefits from the more compact representation, while effects, previously described as benign overfitting [Bartlett et al., 2020], can foster a policy that is robust regarding environmental changes.

Algorithm 1 CROPed Policy Optimization

Formally, we suggest utilizing a reshaping function CROP : S ↦→ S ∗ , where S ∗ is the reshaped observation space, similar to the observation function O in POMDPs. In this paper, we use hard-coded compression functions, but we see great opportunities to extend this work to learning the compression functions as an approach to transfer learning in RL (meta-RL). By using the hard-coded compression function, we can leverage domain knowledge to accelerate learning. For interaction of the agent with the environment, CROP is used as surrogate observation, obtaining the modified observation s∗t = CROP(st), where st is the d-dimensional full observation of the environment at step t. Algorithm 1 demonstrates the application of CROP to an arbitrary policy optimization algorithm.

To illustrate the proposed CROP mechanisms, we show their impact to an exemplary state in the fully observable safety training gridworld (cf. Figure 1d), that is introduced in full detail in section 5. However, it should be noted, that the proposed methods reflect basic concepts to asses the impact of CROP that may be refined, combined, or methodically applied to more complex observations. Concretely we propose the following three CROPs, visualized exemplary in Figure 1:

1. Radius CROP (cf. Figure 1a): Reshapes the

observation to a ρ-sized radius around the agent:

State relevance: ensured by positional proximity. Required Information: the d-dimensional position of the agent α and a padding character to produce consistentsized observations on the edges, as well as an operator s[x : y] reducing the state s to a sub-state between indices x and y.

1. Action CROP (cf. Figure 1b): Reshapes the observation to states accessible via the immediate actions A:

State relevance: ensured by state interactability and

proximity, similar to Radius CROP, but sparser. Additional Information: the agent position α and a set of n action-mutations µ, assigning each possible action a d-dimensional mutation of the agent's position.

1. Object CROP (cf. Figure 1c): Reshapes the full observation (observation of every cell) to the distance vectors from the agent to the nearest η cells for each object type O ⊂ F. Thus, the resulting observation will have the dimension dim(s∗t ) = (|O| · η, d).

State relevance: ensured by object interactability and proximity, comparable to a LiDAR sensor. Additional Information: A scan mechanism σ(s, α, η, o) to find the absolute position of η-nearest cells containing object type o.

All proposed methods transform the observation into a relative state centered/based around the agent, in contrast to the absolute, global, full observation st. Radius CROP and Action CROP can be understood as explicit (hard encoded) attention mechanisms. While all proposed methods transform the given MDP into an POMDP, Radius CROP is the closest resemblance of partial observability in a classical sense within a gridworld scenario, that are typically centered around the agent, especially in arbitrary-sized, potentially infinite environments.

5 Experimental Setup

All implementations for the following evaluations can be found here 1 .

Environments. To provide proof-of-concept for CROP we used two holey safety gridworlds inspired by [Leike et al., 2017] comprising an (7, 9) observation space with a set of five discrete features F = {Wall, Field,Hole,Goal,Agent} and and four discrete actions A = {Up,Right,Down,Left}. To specifically asses the models' robustness to changes in the environment, we train all models in the single training configuration visualized in Figure 2a and test their performance in the unseen distributionally shifted environment shown in Figure 2b. For further evaluation and comparisons in section 7 we use (7, 7)- and (11, 11)-sized generated mazes inspired by [Cobbe et al., 2020] with an identical action space and the reduced set of fields F = {Wall, Field,Goal,Agent} shown in Figure 2c and Figure 2d respectively. Again, to assess generalization, unseen configurations were used to test the trained policies. Therefore, we use a pool of 100 randomly generated mazes explicitly excluding the deterministic configuration to test polices trained in the single maze configurations and to train policies that are tested in the single deterministic environment. The reward range of randomly generated mazes is dependent on the shortest path, thus variable. However, the hardest possible mazes yielding the longest possible shortest paths, result in an optimal reward of 34 and 2 for Maze-7 and -11 respectively. To increase training speed, we trained all policies using four parallel environments.

1 https://github.com/philippaltmann/CROP

Policy Optimization. Whilst applicable to any policy optimization algorithm, we chose to evaluate CROP as an extension to PPO, having shown to be a robust and universally applicable state-of-the-art choice [Schulman et al., 2017]. We furthermore built upon the implementations by [Raffin et al., 2021], extending upon [Brockman et al., 2016]. To analyze the effect of CROP and demonstrate its strengths we provide ablation studies and compare training and evaluation using Full Observations (FO) to all three CROPed observations in section 6. Furthermore we provide benchmark comparisons to a FO Advantage Actor Critic (A2C) from [Raffin et al., 2021] and an alternative method for improved generalization, data augmentation, in section 7.

Data Augmentation (RAD). To provide a state-of-the-art comparison, we implemented data augmentation mechanisms for reinforcement learning (RAD) according to [Laskin et al., 2020] and evaluate their impact in randomly generated mazes in section 7. For a fair comparison to CROP however, we also apply the proposed methods to the discrete full representation of the environment, instead of using images, and use PPO for training as suggested by the authors. Therefore, all image-based transformations like grayscale or color-jitter are not applicable. Thus, we use random-crop, -translate, and - cutout with the same ratios suggested by [Laskin et al., 2020], causing the full observation to be randomly cropped to (6,6) and (9,9), randomly translated to the original shape (7,7) and (11,11), and cut-out by patches sized in the ranges (0,2) and (0,3) in all dimensions.

Hyperparameters. For training PPO, we adopted the default parameters suggested by [Schulman et al., 2017; Raffin et al., 2021]. For the safety environments, we trained all models for a maximum of 1M steps, terminating once 99% of the optimal return (40 for the training environment) is reached. For Radius CROP we set the radius ρ = (2, 2), resulting in an observation shape of dim(s∗t ) = ρ · 2 + 1 = (5, 5), padded with wall fields. Given the four possible actions A = {Up,Right,Down,Left} we parameterized Action CROP with µ = [(-1, 0), (0, 1), (1, 0), (0, -1)], resulting in an observation shape of dim(s∗t ) = |A| = (4). Regarding Object CROP we chose η = 1 for all safety environments and η = 2 for all mazes and the set of objects to be detected to be all possible objects excluding the agent itself: O =

F \ {Agent}, resulting in O = {Wall, F ield, Hole, Goal} and the observation shape dim(s∗t ) = (4, 2) for the train and test environments (cf. Figure 2a and Figure 2b), as well as O = {Wall, F ield, Goal} and the observation shape dim(s∗t ) = (3, 2) for all maze environments (cf. Figure 2c and Figure 2d). All choices above were determined in preliminary experiments, omitted in this work due to limited space.

Metrics. To reflect both the training performance and the generalization capabilities, we regularly (every 213 steps) evaluated the policies during training (without further training) in both the training and an unseen test configuration, reflected by the metrics Validation Return and Evaluation Return respectively. The return is either averaged over 100 nondeterministic episodes (for all maze configurations) or based on a single deterministic episode (for all holey environments, to reflect the certain safety of the current policy). Additionally, to increase significance, all runs are averaged over eight independent seeds.

6 Evaluation

To provide proof of concept, that the reduced information is sufficient for learning an optimal policy, the following section contains ablation studies comparing the performance of PPO for learning a policy in the holey safety training environment (Figure 2a) using Full Observations (FO), Object CROP (O-CROP), Action CROP (A-CROP) and Radius CROP (RCROP) (cf. Figure 1).

The progress of the Validation Return throughout training in the holey safety environment (Figure 2a) is shown in Figure 3a. Overall, all compared approaches find optimal policies, reaching returns above the solution threshold 40, within 150k steps. However, presumably caused by an insufficient representation, finding an optimal policy is the slowest using FOs. On the other hand, the results for all CROPs show, that the compressed information serves sufficient for training an optimal policy. Furthermore, caused by the increased relevance of observed states, the required training steps within the environment are reduced by 50% for R-CROP and O-CROP, compared to FO. The comparably slower training observed for A-CROP is probably caused by its too sparse observation, containing only the four neighboring states, which, again hinders fast convergence.

(b) Evaluation Return in Test Environment However, the real benefits of CROP are exposed in the Evaluation Return shown in Figure 3b. Caused by the previously unseen shift of the positions of the holes in the environments, polices trained using FO only reach returns up to -50. This return is most likely caused by polices that are terminated by a hole, instead of the agent reaching the target state. On the other hand, all CROP-trained polices show to be robust to said shift, resulting in significantly increased Evaluation Returns. Comparing the Evaluation Return with the Validation Returns in Figure 3a shows that said robustness is obtained in parallel to learning to solve the training environment, manifesting the advantages of CROP. Furthermore, all CROP trained polices reach Evaluation Returns above the solution threshold of the test environment of 38, resulting not only in a behavior that is able to navigate to the target state, but also uses the shortest possible path.

Figure 3: CROP Evaluation: Comparing Radius- (blue), Action- (light blue), Object-CROP (green), and Full Observability (red) in the Distributional Shift Safety Environment. The number of steps taken in the environment is on the x-axis and the Validation Return (solid lines in Figure 3a), and Evaluation Return (dashed lines in Figure 3a) on the y-axis averaged over eight random seeds. The shaded areas mark the 95% confidence intervals. The reward threshold of 40 is displayed by the dotted line.

Figure 5a and Figure 5b provide further insights into the resulting policies, showing a heatmap visualization of the dominant (deterministic) action in each state of the unseen test environment, chosen by the FO- and R-CROP-trained policies respectively. Heatmaps depicting A-CROP and O-CROP policies are omitted, as their behavior resemble the results shown for R-CROP. As assumed above FO-trained policies reveal a behavior directly navigating into the nearest hole, even though the policy has learned to evade the holes in the training environment (cf. Figure 5a). This behavior suggest that the trained policies overfit to the full observation of the training environment. The polices trained using RCROP on the other hand are able to evade the shifted holes, even though, the environment has not been seen during training (cf. Figure 5b). Furthermore, the heatmap reveals, that the trained policy is able to reach the target within the shortest possible path, from any position, even though, the training was only conducted with the agent starting in the top left field. This generalization capability can not be observed for the FO-trained policies at all, only reaching the target from the neighboring field and otherwise failing to fulfill the intended task.

Overall, the evaluations results provide evidence that CROP reduces the information to an efficient representation containing the important information for finding an optimal policy, whilst accelerating training performance and improving robustness to distributional shifts by the removal of unimportant details, otherwise prone to overfitting.

7 Benchmark Comparison

To further asses the generalization capabilities of CROPtrained policies, we provide comparisons to the Full Observation (FO) for training both PPO and A2C and Augmented Observations (RAD) for training PPO in three increasingly complex maze settings. As suggested in [Cobbe et al., 2020], we first evaluate generalization ability when training for 200k steps using a pool of randomly generated configurations.

The Validation Return is shown in Figure 4a. Similar to the previous evaluations, all observation types provide sufficient information for learning an expedient policy using PPO. However, both FO-A2C and RAD only reach performances below -20, while both FO-PPO and R-CROP reach near optimal performance of around 40. Again, presumably caused by the compressed observation, R-CROP shows the fastest convergence within about 100k steps, where FO and O-CROP increase the required steps by about 50%, A-CROP also provides sufficient information to reach the target, but its sparsity seems to hinder convergence to an optimal behavior.

However, evaluating the polices' generalization capabilities by analyzing their performance in the single unseen Maze-7 configuration, shown in Figure 4b, all CROP-trained polices show near optimal performance, with final Evaluation Returns of A-CROP, R-CROP, and O-CROP around 40. On the other hand, FO-trained policies initially show good generalization to the unseen test configuration within the first 100k steps. However, as the trained policy explores a near optimal solution in the training configuration, reaching Validation Returns upwards of 20, generalization drastically decreases and nearly drops to the minimum of -100 after 200k steps, indicating evaluation episodes where the final policy is not able to reach the target at all. This artifact of overfitting not showing for all CROP methods again confirms above assumptions regarding benign overfitting, thus improved generalization, encouraged by the improved observation. Interestingly, both FO-A2C and RAD-PPO show Evaluation Returns similar to CROP, for the single maze configuration, indicating either improved generalization, or, just less overfitting, given their Validation Return performed worst in comparison.

Increasing the maze size to 11 whilst relaxing the training complexity by using only a single configuration, similar effects show (cf. Figure 4c). All CROP-trained, as well as the FO-PPO trained policies reach near optimal performance within about 140k steps, while R-CROP and O-CROP show the fastest convergence. FO-A2C- and RAD-PPO-trained polices show the worst performance around the minimal return of -100. Interestingly however, even though showing slightly sub-optimal training performance of around 15, ACROP reaches the highest Evaluation Returns, thus best generalization. Note, that in this scenario, generalizing from a single training environment to 100 random unseen test environments is a notably harder challenge compared to the previous experiments, explaining the considerably lower overall Evaluation Returns. Nevertheless, while FO- and O-CROPtrained policies show indications of overfitting to the training environment in their Evaluation Return, R-CROP and ACROP trained policies generalize to a behavior at least reaching the goal within unseen maze configurations. FO-A2Cand RAD-PPO-trained polices merely reach the target of the test environments at all, which is unsurprising given their training performance.

Finally, Figure 4e shows the Validation Return training in 100 randomMaze-11 configurations. In contrast to the previous results, only Radius CROP-trained policies reach performances above -50, with the final return -10. Remarking the theoretical worst case maximum return for generated Maze11 configurations of 2, however, this translates to an 90% optimal behavior. Furthermore, as for all previous experiments, the agent is not able to observe the target from the initial position using R-CROP, making this high performance even more remarkable. Moreover, this also translates to the Evaluation Return shown in Figure 4f, where a near optimal performance of around 20 is reached. Generally, due to their increased complexity, larger mazes have shown to be less prune to overfitting, especially when training with a pool of random configurations. However, still, CROP trained policies, especially using the proposed Radius method show the fastest training convergence and the best generalizing capabilities to unseen configurations.

8 Conclusion

Overall, we formalized a method for Compact Reshaped Observation Processing (CROP) and proposed three concrete CROPs applicable to fully observable discrete environments: Radius CROP, compressing the observation w.r.t. close positional proximity, Action CROP, compressing the observation w.r.t. interactability, and Object CROP, compressing the observation w.r.t. relations to surrounding objects. Furthermore, we showed the improvement using any of the proposed CROP over the full observation regarding both the training speed (steps until an optimal policy is found) and, more importantly, the robustness to a distributional shift in the environment in two holey safety environments. Finally, benchmark comparisons to the state-of-the-art using augmented data in procedural generated mazes further confirmed the advantages of CROPed observations, showing improved generalization to unseen maze configurations. Overall, Radius CROP has shown most beneficial, outperforming the full observation in all tested configurations.

Overall, we believe that CROPing observations to improve their information relevance is a promising approach for improving both the robustness and reliability of reinforcement learning algorithms, by preventing overfitting to unimportant details in the environment.

Future work should therefore consider methods and applicability to further observation spaces. Regarding Action CROP, safety critical domains where actions may be constrained due to potential catastrophic failures, could be examined. Also, further CROP methods could be developed with attention to different important features in the observation space. Furthermore, the proposed concepts could be applied for training a CROP mechanism for automatic state reduction. For example, they could serve as a target for the latent space of an autoencoder, that could be applied to CROP continuous or even partially observable observation spaces. Generally, various meta learning concepts may be applied to extend CROP for a more universal applicability.

Acknowledgments

This work was partially funded by the Bavarian Ministry for Economic Affairs, Regional Development and Energy as part of a project to support the thematic development of the Institute for Cognitive Systems and is part of the Munich Quantum Valley, which is supported by the Bavarian state government with funds from the Hightech Agenda Bayern Plus.