IntEr-HRI Competition: Intrinsic Error Evaluation during Human - Robot Interaction

1 Introduction

EEG-based detection of human intentions and their dynamic changes have gained significant importance in human-robot interaction (HRI). It has been thoroughly studied in braincomputer interface (BCI) applications, especially in robot learning and human-robot co-adaptation. In particular, errorrelated potentials (ErrPs) evoked in the brain when observing erroneous actions of other humans or systems, such as robots, have been extensively studied across various research areas (see [Chavarriaga et al., 2014] for a comprehensive review). Such studies enable the decoding of human intentions using robots, thereby enhancing communication with humans and improving behavioral strategies based on real-time updates (e.g., [Kim et al., 2017], [Kim et al., 2020]).

In most ErrP-based applications, erroneous behavior of robots was detected with the help of visual cues (e.g., [Iturrate et al., 2010],[Kim and Kirchner, 2013], [Kim and Kirchner, 2015]). However, certain studies have used visuo-tactile stimuli to evoke ErrPs. These studies either integrated both visual and tactile channels to perceive erroneous behaviors of systems ([Tessadori et al., 2017], [Schiatti et al., 2019]) or used the visual channel to recognize errors and the tactile channel (e.g., vibration) solely to indicate upcoming errors([Perrin et al., 2008], [Chavarriaga et al., 2012], [Ahkami and Ghassemi, 2021]). Moreover, to the best of our knowledge, only tactile-based ErrP detections have not been investigated yet. Thus, this competition served as a means to encourage systematic investigation of tactile-based ErrP detection in HRI using machine learning approaches.

Furthermore, there were a few competitions that recorded openly accessible EEG datasets where specific brain patterns in the time or frequency domain were evoked synchronously in response to, for example, motor imagery or workload without the use of robots (e.g., [Roy et al., 2022], [Blankertz, 2004], [Blankertz, 2008]). There are also some open EEG datasets where ErrPs were evoked synchronously by visual stimuli from observing robots' incorrect behavior (e.g., [Ehrlich and Cheng, 2019]). However, there are no open-access datasets where robots are used to evoke ErrPs tactilely.

Thus, to address this gap, for this competition, we developed an HRI scenario in which subjects tactilely perceived the incorrect behavior of an active orthosis device (see [Kueper et al., 2024]) during the execution of arm movements (flexions and extensions). The recorded open source EEG and EMG dataset (see Section 2.1) from the offline stage of the competition is accessible on Zenodo 1 . This dataset can be used to further research the asynchronous detection of tactilely evoked ErrPs, which was also the goal of our competition.

This paper is structured as follows: Section 2 describes the task and structure of the competition. In Section 3, we provide details on the evaluation metrics used to determine competition winners, while Section 4 presents an overview of the results. Finally, Section 5 discusses the key takeaways and insights provided by the competition.

1 https://zenodo.org/records/8345429

2 Task and Structure of the Competition

As described in Section 1, the competition's goal was to encourage the scientific community to contribute towards improving the online classification of tactile-based error onsets in EEG data. Thus, the participating teams were tasked with developing robust and reliable signal processing and machine learning approaches to detect erroneous behaviors through single-trial EEG analysis asynchronously.

The competition comprised two stages - offline stage and online stage, and only the top-performing teams from the offline stage progressed to the online stage. The winners of the competition were announced at the IJCAI 2023 conference. During the conference, we also organized five keynote lectures (available on our competition homepage 2 ), a live panel discussion and paper presentations by participating teams.

2.1 Offline Stage of Competition

In the offline stage, we recorded the first open-access EEG and EMG dataset 1 , wherein tactile errors were deliberately introduced via an active orthosis device (see [Kueper et al., 2024] for detailed information). Eight healthy subjects participated in the study, and 10 data sets were recorded, each consisting of 30 flexion or extension movement trials. Six trials in each set were randomly chosen for introducing tactile errors for a duration of 250 ms. Whenever the subjects felt an error, they were instructed to press an air-filled ball in their left hand (direct response), and these events were marked in the labeled EEG data.

The challenge consisted of a training phase and a testing phase. During the training phase, participants accessed only eight of the 10 recorded datasets, each containing labeled EEG data, to train their classifiers in detecting error onsets (six errors in each set). Furthermore, teams validated their models via 10-fold cross-validation and submitted results in a short paper (accessible on our competition homepage 2 ). Subsequently, in the testing phase, the remaining two sets were provided as unlabeled EEG data, and the teams tested their pre-trained models offline and submitted error onset prediction results.

2.2 Online Stage of Competition

During the online stage, unlabeled EEG data was streamed via the Lab Streaming Layer (LSL) ([Kothe et al., 2018]) with the help of a VPN tunnel. The challenge was to employ a pre-trained classifier for real-time error onset detection in the streamed EEG data. As we wanted to evaluate the robustness of the model in real-world session transfer scenarios, no calibration phase was included. Instead, a dry run (without real data) was conducted a day before the competition to test live data streaming through LSL and to ensure the general functionality of the teams' code.

On the competition day, a subject from the offline stage was prepared for the experiment, and the teams were notified about the subject in advance. The online experiment consisted of two cases: direct response case and no response case. The direct response case involved the subject pressing an air-filled ball upon recognizing an error, whereas in the latter case, the subject refrained from any response. These cases were introduced to evaluate the impact of motor-related activity in the EEG in real-world scenarios. Furthermore, each case included two sets of 30 movement trials wherein five trials were randomly selected to deliberately introduce tactile errors lasting 250 ms.

2 https://ijcai-23.dfki-bremen.de/competitions/inter-hri/

3 Performance Evaluation

The winners of the competition were decided through a quantitative performance evaluation scheme as detailed on our competition home page 2 . Each of the two stages had different evaluation metrics, as described below.

3.1 Offline Stage Evaluation

In the offline stage, as mentioned in Section 2.1, each team tested their classifier models on the 16 test sets (8 subjects; 2 test sets per subject) and submitted the sample indices corresponding to the predicted error onsets. Each submitted sample index was compared against reference ground truth to calculate the time error. The accumulated error score for the team was the sum of all individual time errors. If the error was negative (early prediction) or greater than 1000 ms (late detection), it was classified as a false positive. Each false classification incurred a penalty of 1000 ms. This method was designed to reward timely detection of the robot's erroneous behavior, which is critical when using assistive robotic devices.

3.2 Online Stage Evaluation

To determine the winners of the online stage, we devised the Final Performance Score (FPS). This novel quantitative metric combines balanced accuracy (BAcc) with a new metric termed time score (tscore). This evaluation was based on two critical aspects: accurate error onset prediction and realtime classification of errors. Both these aspects were quantified into a single parameter known as the individual sample score using logistic regression. Thus, each set's time score (tscore) was computed by summing up all the individual sample scores and normalizing them between 0 and 1. Finally, the FPS was calculated by assigning a weightage of 70% to BAcc and 30% to tscore. The closer the FPS is to 1, the better the trained classifier's ability to accurately classify tactile-based error onsets in real time with minimal computation delay.

4 Competition Results

As discussed in Section 2, this competition was designed to allow only the best-performing teams from the offline stage to participate in the online stage. This section presents the results of both stages of the competition. Note that the names and affiliations of the top-performing teams are mentioned in the demonstration video (https://youtu.be/JFC-Kc3FHEc).

4.1 Results for the Offline Stage

A total of six teams had registered for the offline stage, out of which only three teams could solve the task and qualify for the online stage. The offline stage results of these three teams are provided in Figure 1.

As the accumulated error score was the summation of time errors across all the sets, a lower score was more desirable. Thus, team A was the clear winner of the offline stage, followed by team B and team C.

4.2 Results for the Online Stage

Of the three teams that qualified for the online stage, only one team (team B) could detect and classify errors in real time. Its performance was evaluated per the metrics described in Section 3.2 and provided in Figure 2. The arithmetic mean of all four individual FPS gave the final overall performance metric for the online classifier model, which was 0.805 on a scale of [0,1].

5 Key Insights

This competition proved to be a success for varied reasons. In addition to providing valuable technical insights to the participating teams and the organizers, it successfully demonstrated the challenges of asynchronous online classification of errorrelated activities from the human EEG.

Despite the top three teams having an average validation BAcc score of around 90% in the offline stage, their performance dropped greatly in the online case. Two of the three finalists failed to detect a single error, and the winning team scored an average BAcc of less than 80%. The main reason for this discrepancy lies in the low signal-to-noise ratio (SNR) and high variability of EEG signals across different sessions, even for the same subject. These inherent properties of EEG make the development of a robust machine-learning model quite challenging (see [Wu et al., 2022] for a comprehensive review).

We also encountered several challenges while organizing this competition. The most significant among them was

(A) Direct Response

(B) No Response

designing the experiment to evoke an ErrP through robotinduced tactile error introduction. Additionally, it was essential to design a framework that could stream live EEG data with minimal communication delay and near-zero sample loss for the online stage. This framework was created by combining a VPN tunnel and the LSL protocol as mentioned in Section 2.2. Furthermore, we developed a novel performance evaluation metric that could be used for benchmarking asynchronous online EEG classifiers. This metric considers key evaluation aspects, such as accurate prediction, compensation of class imbalance, and computation time, that determine how well the asynchronous detection works. Moreover, we integrated this metric with our live-streaming framework to ensure that the communication delays and sample losses did not impact the comparison of teams' online performances.

6 Conclusion

The IntEr-HRI competition at IJCAI'23 addressed the challenges associated with the asynchronous detection of intrinsic feedback using EEG signals. A specific HRI scenario was designed to evoke ErrPs in the human EEG through robotinduced tactile stimuli. The scientific community was encouraged to develop state-of-the-art machine learning models to detect erroneous behavior of the robotic device continuously. The competition was structured considering real-world application scenarios, and special emphasis was placed on robustness and reliability in the evaluation metrics. To stimulate further research, an open-access EEG and EMG dataset was made available, alongside a detailed account of experimental design and methodology in [Kueper et al., 2024]. We hope that through this competition, we could demonstrate research gaps in the online detection of tactile-based ErrP and direct the scientific community's attention towards further improving assistive technology using psychophysiological data such as EEG data.

Ethical Statement

The studies involving human subjects were approved by the Ethics Committee of the Department of Computer Science and Applied Cognitive Science of the Faculty of Engineering at the University of Duisburg-Essen. The studies were conducted following the local legislation and institutional requirements. The participants provided their written informed consent to participate in the study. Written informed consent was also obtained from the individual(s) for the publication of any potentially identifiable images or data included in this paper or the video.

Acknowledgements

We sincerely thank our colleagues, Judith B¨utef¨ur and Julia Habenicht, for their invaluable assistance in conducting the studies. Additionally, we acknowledge the contributions of Tobias Rossol, Marc Tabie, and Jan-Philipp Brettschneider in the mechanical design and setup of the active orthosis device. We would also like to thank Mathias Trampler for his support in developing the live data streaming solution.