On Adaptivity and Safety in Sequential Decision Making

1 Introduction

Sequential decision making systems are all around us, from recommendation systems used by Amazon, Netflix, etc for personalized recommendations to online advertising done on platforms like Google or Bing. Even robot learning through environment or simulator access can also be cast as sequential decision making problem. More recently, conversational agents like ChatGPT employ sequential decision making to understand the context and the history of a particular conversation to provide more accurate responses.

Particular instances of decision making algorithms are employed to solve these practical systems. Contextual bandits, for example, are used in recommendation systems to select the most relevant content or products to recommend to users based on their previous behavior and preferences. Depending on the robotic application at hand, model predictive control or reinforcement learning are used to optimize for robot control policies. And GPT-3 is adapted to dialogue format, giving ChatGPT, using reinforcement learning from human feedback.

We focus on two important challenges that arise in these practical systems, namely need for safety and adaptability. We consider adaptability in the further challenging setting of sparse reward feedback. We study these challenges through algorithmic paradigms of reinforcement learning (RL), and online convex optimization (OCO).

Adaptability In the real-world applications, it is not enough to train an agent to perform a single task or operate in a specific environment. Instead, agents must be able to generalize their knowledge and adapt to new situations. For example, in robotics, a robot may need to navigate to new environments or perform new tasks.

Sparse reward or feedback Most formulations of sequential decision making rely on learning a meaningful agent model/policy through means of a feedback signal called cost/reward function. Most real-world applications, however, do not provide a feedback at every time instance in the decision making process. Such sparsity in feedback is detrimental to the learning of agent model.

Safety Inmany real-world applications, the actions selected by the decision maker must satisfy some necessary safety constraints over the decision set. For example, in robotics applications, the control actions should maintain the closedloop stability of the system. Typically, such constraints are represented using a safe decision set definition. The control action then must lie inside this safe decision set for safe operation of the system.

2 Background

Reinforcement Learning and Meta Reinforcement Learn-

ing Reinforcement learning (RL) is an intuitive way to model learning through interaction. Traditional RL algorithms require a significant amount of training data to learn a single task or operate in a specific environment. On the contrary, in many real-world applications, the environment is inherently uncertain and dynamic, and agents need to be able to adapt quickly to new situations. Adapting to such environment changes quickly may require agents to learn from a small amount of data, which can be extremely challenging.

Meta reinforcement learning (meta-RL), a rapidly growing area of research, addresses this challenge of quick and efficient adaptability by enabling agents to learn how to learn. Meta-RL allows agents to develop a set of meta-policies or meta-knowledge that can be used to guide their learning process and adapt to new situations quickly.

Online Convex Optimization Online convex optimization (OCO) models sequential decision making only through the action and the respective cost function sequences. Here, an agent repeatedly makes decisions based on the feedback received from the environment. Agent's goal is to minimize cumulative cost with respect to the best action in hindsight, a quantity commonly referred to as 'Regret'. Simply put, OCO can be considered as a stateless version of RL, and is studied to understand a problem with its probable solution approaches in a simpler setting.

3 Contributions

Inspired from the need for adaptability and safety mentioned in Sec. 1, we have devised algorithms (1) that perform model adaptation using sub-optimal demonstration data in sparsereward environments with meta reinforcement learning , and (2) that respect safety constraints at every time step in online convex optimization.

Meta-RL in sparse reward environments Meta-RL is a powerful approach for solving real-world problems [Finn et al., 2017], but it faces a significant challenge when reward functions are sparsely specified, meaning that feedback values are only available for certain decisions. To address this issue, we investigate the use of sub-optimal demonstration data that is available for each task. We propose an algorithm called Enhanced Meta-RL using Demonstrations (EMRLD) [Rengarajan et al., 2022] that leverages the demonstration data to provide a proxy for missing reward feedback during training. EMRLD combines reinforcement learning and supervised learning over demonstration data to generate a metapolicy that improves performance in a monotonic fashion. We conducted experiments on various sparse reward environments, including a mobile robot, and found that our EMRLD algorithm significantly outperforms existing approaches.

OCO with unknown linear constraints We consider the problem of safe online convex optimization, where we need to choose an action at each time step that satisfies a set of linear safety constraints, even though we don't know the specific parameters that define these constraints. We can only observe noisy feedback about the constraints for the actions we choose. Our proposed algorithm, called SO-PGD [Chaudhary and Kalathil, 2022], achieves a regret of O(T 2/3√log(T)) if we have access to a safe baseline action. This means that we can optimize our actions without violating the safety constraints. We have also developed algorithms to ensure safety in multi-agent settings with unknown linear safety constraints using distributed-OCO [Chang et al., 2023]. For convex loss functions, our algorithm achieves a dynamic regret of O(T 2/3√log(T ) + T 1/3C ∗ T ), where C ∗ T is the length of the best minimizer sequence. For certain non-convex problems, we achieve a dynamic regret of O(T 2/3√log(T )+T 1/3C ∗ T ).

In addition, we have explored a relaxed form of safety that involves smooth policies in imitation learning [Chaudhary and Ravindran, 2022]. This is an important step towards extending our results to high-dimensional problems.

4 Conclusion and FutureWork

Our ultimate goal is to develop safe and efficient RL algorithms that can learn from experience and adapt to changing environments while ensuring stability and avoiding dangerous or unintended behaviors. In this regard, we are further exploring new ways to address these challenges using the context-based meta-RL approaches. Context-based metaRL broadly encompasses probabilistic meta-RL [Rakelly et al., 2019] and Bayesian meta-RL [Zintgraf et al., 2019; Dorfman et al., 2020], methods that reason about uncertainty in the RL system and make more informed decisions.