Q-Learning-Based Model Predictive Variable Impedance Control for Physical Human-Robot Collaboration (Extended Abstract)

1 Introduction

To meet customers' needs, which are becoming more and more oriented on tailor-made products, companies are updating their production processes by means of new flexible and agile tools [Fragapane et al., 2020]. In this context, collaborative robotics plays a key role [Makris, 2021], providing powerful solutions to assist the operators in the execution of different activities, such as co-manipulation [Roveda et al., 2018a; Roveda et al., 2019], task's knowledge transfer to the robotic system [Roveda et al., 2021b; Roveda et al., 2021a], easily programmable and deployable applications [Vicentini et al., 2020], etc. Physical humanrobot collaboration (pHRC) is currently one of the most investigated topics [Galin and Meshcheryakov, 2020]. In fact, pHRC is nowadays demanded in many fields of applications, both for collaborative robots [Roveda et al., 2018b] and exoskeletons [Mauri et al., 2019]. However, many open issues in the state of the art are still to be overcome, in particular considering safety/stability guarantees in the human-robot interaction, human-robot dynamics modeling, human intention recognition (for active assistance/empowering purposes), and computation efficiency (for real-time control adaptation and optimization).

∗This work is described in the paper [Roveda et al., 2022] Loris Roveda, Andrea Testa, Asad Ali Shahid, Francesco Braghin, and Dario Piga. Q-learning-based model predictive variable impedance control for physical human-robot collaboration. Artificial Intelligence, 312:103771, 2022.

To tackle the above-mentioned issues within the pHRC scenario, this paper proposes a Q-Learning-based Model Predictive Variable Impedance Control (Q-LMPVIC) to assist the operator while physically interacting with a collaborative robot. Based on Cartesian impedance control (providing the controlled manipulator a compliant and decoupled behavior in the Cartesian space), an MPC is designed in order to online optimize its parameters (i.e., setpoint and damping parameters) to assist the user along the detected intended motion direction(s), maximizing the collaboration performance. The MPC exploits a learned human-robot interaction dynamics model, obtained by means of an ensemble of neural networks. Therefore, the lack of sophisticated analytical models for the human-robot interaction dynamics is overcome, by employing a method that is capable to capture the complexity and uncertainties of such dynamics. An MPC objective function is designed in order to minimize the user's effort during the collaboration with the robot. Indeed, the user's intention of motion can be detected, making it possible to assist him/her along the intended direction(s) of motion. The designed MPC is also enhanced with stability guarantees by means of Lyapunov constraints. In such a way, safety/stability issues are tackled by the proposed methodology. The MPC is then (online) solved by making use of a Q-Learning method, exploit-

.

ing an actor-critic algorithm to approximate its exact solution. The obtained solution is accurate and highly efficient, being able to tackle the issue related to computation efficiency that might compromise the implementation of the controller for real applications. The proposed approach has been applied to two complex use cases (a collaborative assembly task and a collaborative deposition task) in order to show its applicability to real industrial tasks.

While other approaches have been developed to deal with the proposed topic (i.e., pHRC) [Roveda et al., 2020; Cremer et al., 2019; Roveda et al., 2019; Dimeas and Aspragathos, 2015; Medina et al., 2019; Peternel et al., 2017; Zhang et al., 2020], they are indeed characterized by some difficulties in the simultaneous optimization of the impedance parameters. The online optimization of the setpoint and the other impedance parameters is, in fact, important to obtain an active target-oriented and compliant behavior of the manipulator during pHRC task. In addition, it is commonly difficult to ensure stability in such an optimization problem. Moreover, a reliable model of the target dynamics is not always available, making it difficult to optimize the collaboration. Besides, most of the strategies are based on the internal simulation of the state evolution, exploiting the prediction capabilities of dynamic models or Q-functions. However, the former strategies are often not enough accurate or efficient to be implemented for real applications execution, and the latter ones are usually approximated with fuzzy logic, requiring a precise setting of the fuzzy rules and membership functions to solve the "curse of dimensionality" problems.

2 Methodology

The proposed Q-Learning-based Model Predictive Variable Impedance Control (Q-LMPVIC) for pHRC tasks is made up of two main levels. In the first one (i.e., the low-level control loop), a variable Cartesian impedance controller is realized, such that the outer high-level controller could work considering the manipulator as a decoupled mass-spring-damper system in the Cartesian space. The outer high-level controller is then used to update the setpoint and the damping parameters of the inner controller in order to optimize the pHRC performance (i.e., minimize the interaction force between the human and the robot, and, therefore, the operator's effort). The outer high-level controller is composed of an actor and a critic ANN, which implements a Q-Learning algorithm for the resolution of a nonlinear optimal control problem. An ensemble of ANNs is exploited to estimate the model of the system. An MPC enhanced with stability guarantees (by means of Lyapunov constraints) is implemented to online compute the low-level impedance control parameters, maximizing the pHRC performance.

Figure 1 shows the proposed Q-LMPVIC schema, highlighting each element composing the proposed methodology. In the following, all the elements composing the Q-LMPVIC (i.e., the low-level Cartesian impedance control, the human-robot interaction dynamics, the modeling estimation methodology based on the ensemble of ANNs, the L-MPC, the Q-learning methodology, its actor-critic ANNs, and the CEM algorithm) are described.

Remark 1. All the theoretical derivation of the approach is available in the authors' work [Roveda et al., 2022].

3 Experimental Validation

The developed controller has been employed in two industrial tasks to provide insights into its usage in real production scenarios. In particular, (i) a collaborative assembly task of a gear into its shaft [Roveda et al., 2021b] (Figure 2 (a)) and (ii) a collaborative deposition task [Roveda et al., 2021a] (Figure 2 (b)) have been considered as target use-cases. (i) considers the robot equipped with a gripper manipulating a gear to be collaboratively assembled. (ii) considers the robot equipped with a sealing gun operated by the human along the deposition path, strictly related to the activities developed in the H2020 CS2 ASSASSINN project. Both tasks can be executed to demonstrate the application to the robot (e.g., in the context of programming-by-demonstration) or to collaboratively perform it.

The selected tasks allow the assessment of the adoption of the proposed controller in real human-robot collaborative industrial tasks. In fact, such tasks require a smooth humanrobot interaction, having the robot capable of quickly reacting to the intention of motion of the human in order to properly execute the target task, guaranteeing the stability of the interaction. In fact, instabilities, vibrations, or delays in the controller reaction might cause task failures or unacceptable output quality. Indeed, the proposed use-cases allow the evaluation of the effectiveness of the developed controller in complex human-robot collaborative applications.

A video showing the performed assembly task is available at the link, while a video showing the performed deposition task is available at this link. As can be highlighted, the proposed controller allows the implementation of a smooth and reactive robot behavior (reflecting the performance evaluation provided in the previous Sections), making it possible to perform complex human-robot collaborative tasks. Thus, the proposed controller is proven to be applicable to real industrial tasks.

Remark 2. The complete experimental analysis of the developed approach is available in the authors' work [Roveda et al., 2022], where the performance of the proposed controller are compared with the ones achieved by [Roveda et al., 2020].

4 Conclusions

The presented work in this paper proposes a Q-Learningbased Model Predictive Variable Impedance Control QLMPVIC to assist the operators in physical human-robot collaboration (pHRC) tasks. The proposed methodology is composed of two control loops, a low-level Cartesian impedance controller, and a high-level methodology for the online optimization of impedance control parameters (i.e., the setpoint and damping parameters), allowing to minimize the humanrobot interaction force during the collaboration. The outer high-level controller is composed of an actor and a critic ANN, which implement a Q-Learning algorithm for the resolution of a nonlinear optimal control problem. An ensemble of ANNs is exploited to estimate the model of the system. The MPC enhanced with stability guarantees (by means of Lyapunov constraints) is, indeed, implemented to compute the low-level impedance control parameters online. The proposed Q-LMPVIC has been tested to assess its performance. Two complex use-cases (a collaborative assembly task and a collaborative deposition task) have been setup to show the applicability of the proposed approach to real industrial tasks.

Future work is devoted to increasing the high-level control loop frequency (imposed equal to 6 Hz in this paper) in order to further improve the pHRC performance. To solve this issue, parallelization with modern GPUs can be exploited. In addition, external sensors (such as EMGs) can be exploited to better capture the human-robot interaction modeling and collaboration performance. The proposed strategy will also be adapted (in terms of modeling and control objectives) and applied to an exoskeleton device.

Ethical Statement

There are no ethical issues.

Acknowledgments

The work has been developed within the project ASSASSINN, funded from H2020 CleanSky 2 under grant agreement n. 886977.

The work has been developed within the project ExoRescue, funded from Eureka Eurostars under grant agreement n. E!115182.