Benchmarking Stroke Forecasting with Stroke-Level Badminton Dataset

1 Introduction

Sports analytics has garnered increasing attention since the advancement of technology, which greatly facilitates data collection and increases data diversity. In recent years, there has been a surge in studies applying advanced artificial intelligence techniques, e.g., computer vision on video frames [Kim et al., 2022], and machine learning models for action valuing [Decroos et al., 2019; Merhej et al., 2021]. Badminton, as one of the major racket sports worldwide in terms of participation, demands high physical and tactical conditions, attracting researchers to introduce novel applications

[Wang et al., 2020]. For instance, [Wang et al., 2022a] propose long short-term extractors to quantify the win influence of each shot within a rally, while [Chang et al., 2023] design the movement forecasting task to predict players' movements using graph-based approaches.

In this paper, our aim is to introduce, ShuttleSet22, a stroke-level badminton singles dataset collected from realworld high-ranking matches in 2022. ShuttleSet22 extends the original ShuttleSet [Wang et al., 2023], comprising 30,172 strokes (2,888 rallies), 1,400 strokes (450 rallies) in the validation set, and 2,040 strokes (654 rallies) in the testing set. While ShuttleSet22 shares similar stroke-level data formats with ShuttleSet, it consists of matches in 2022 instead of the period between 2018 and 2021. Therefore, ShuttleSet22 can be considered the most recent iteration of the badminton singles dataset, enabling the examination of model effectiveness in recent matches. ShuttleSet22 is sourced from public videos 1 and has been meticulously labeled by domain experts with the shot-by-shot labeling tool [Huang et al., 2022].

To boost researchers' engagements in badminton analytics, we have, for the first time, initiated a challenge within CoachAI Badminton Challenge 2023 2 in conjunction with IJCAI 2023. Specifically, we have organized the forecasting of future turn-based strokes in badminton rallies (Track 2) and provided the state-of-the-art stroke forecasting approach [Wang et al., 2022b] as the official baseline. This track has attracted approximately 100 participants aiming to improve the effectiveness of the stroke forecasting task, with 16 teams submitting their final results on the leaderboard. In addition to the results, we summarize these insights and analyses of this challenge to inspire ideas for bridging the gap between badminton analytics and artificial intelligence communities 3 .

1 http://bwf.tv/

2 https://sites.google.com/view/coachai-challenge-2023/

3 The video can be found at https://youtu.be/yhRouMpxb2M.

2 RelatedWork: Public Badminton Datasets

Generally, there are only a few public badminton datasets due to the heavy cost of collecting and labeling fine-grained records by domain experts [Wang, 2022]. Recently, researchers have released badminton datasets to foster the sports community. For instance, the shuttlecock datasets [Cartron, 2022] contains 8K images of shuttlecocks which are resized to 640*640 pixels. This dataset includes the position of shuttlecocks which could train an object detection model. The BadmintonDB [Ban et al., 2022] features rally, strokes, and outcome annotations between two players, which can be used in player-specific match analysis and prediction tasks, which is also the stroke-level dataset. However, BadmintonDB only consists of the same matchup (i.e., Kento Momota and Anthony Sinisuka Ginting) instead of the various matchups. Moreover, BadmintonDB collects matches from 2018 to 2020, while our ShuttleSet22 collects high-ranking matches in 2022 to reflect the state-of-the-art tactic records. We summarize the discrepancies between our ShuttleSet and BadmintonDB in Table 1.

3 The Challenge System

Conventional applications in badminton mainly focus on quantifying stroke performance [Wang et al., 2021] or retrieving information from videos [Chu and Situmeang, 2017], which motivates us to further investigate a more challenging yet critical real-world application. The goal of this shared task is to forecasting future turn-based strokes in badminton rallies, which aims to design forecasting models capable of predicting future strokes, including shot types and locations, based on past stroke sequences. Figure 1 illustrates the entire flowchart of our task. As we have built the pipeline from data processing to evaluations, participants only need to modify the stroke forecasting model to swiftly iterate their approaches. The task page is available on Codalab.

Figure 2: Heatmap of player and opponent locations in ShuttleSet22.

3.1 ProblemFormulation

Following the definition outlined in [Wang et al., 2022b], for each singles rally, given the observed τ strokes with type-area pairs and two players, the goal is to predict the future strokes, including shot types and area coordinates, for the next n steps. To simplify the problem, τ is set to be 4 in our challenge, and n is various based on the length of the rally, which is given to the participants.

Let R = {Sr, Pr}|R| r=1 denote historical rallies of badminton matches, where the r-th rally is composed of a stroke sequence with type-area pairs Sr = (⟨s1, a1⟩, · · · , ⟨s|Sr|, a|Sr|⟩) and a player sequence Pr = (p1, · · · , p|Sr|). At the i-th stroke, si represents the shot type, ai = ⟨xi, yi⟩ ∈ R2 are the coordinates of the shuttle destinations, and pi is the player who hits the shuttle. We denote Player A as the served player and Player B as the other for each rally in this paper. For instance, given a singles rally between Player A and Player B, Pr may become (A,B, · · · ,A,B). We formulate the problem of stroke forecasting as follows. For each rally, given the observed τ strokes (⟨si, ai⟩)τi=1 with players (pi)τi=1 , the goal is to predict the future strokes including shot types and area coordinates for the next n steps, i.e., (⟨si, ai⟩)τ+n i=τ+1 . We note that n is pre-defined as the actual length of the corresponding rally.

3.2 Exploratory Data Analysis

We have released 58 matches, approximately 4,000 rallies with shot-level records following the BLSR format [Wang et al., 2022a] for this task, where parts of the matches from 2018 to 2021 are identical to those in [Wang et al., 2022b], while we have further included new matches collected from 2022.

Figure 2 shows the heatmap of player location and opponent location, indicating that most players position themselves at both the center and the four corners of the courts when preparing to attack. On the contrary, when defending, players intend to stay predominantly in the center of the court to enable quick reactions to any type of shot.

Table 2 presents the numbers of different strike orders by shot types. It is evident that with the exception of the Clear and Smash shots, other shot types do not target around the head area. Furthermore, more than half of the Push/Rush, Defensive Shot, and Short Service shots are executed with the backhand, while other shot types predominantly utilize the forehand. Notably, there is a significant disparity in hitting position based on the service type, with almost all short services directed below the net and almost all long services aimed above the net.

3.3 EvaluationMetrics

The evaluation scenarios will be assessed by cross-entropy for shot type prediction and mean absolute error (MAE) for area coordinates prediction, similar to the original work. Given the stochastic nature of this task, each team will be required to generate 6 predicted sequences for each rally, from which the closest one to the ground truth will be selected for evaluation. It is worth noting that the original work on the stroke forecasting task involved generating 10 sequences, a number we have reduced to 6 for efficiency.

3.4 Official Baseline

Generally, ShuttleNet [Wang et al., 2022b] is a positionaware fusion of rally progress and player styles framework consisting of Transformer-based architectures. ShuttleNet has demonstrated superior performance in predicting the next strokes compared to conventional sequential. This is attributed to its turn-based architecture, which separates the styles of both players in a rally and integrates them with the current rally condition.

4 Participating System

Approximately 95 participants have joined the challenge system, with 16 teams submitting their testing results for the final phase of the CoachAI Badminton Challenge 2023 (Track 2). Most teams modified the official baseline, ShuttleNet, to address the task. The brief descriptions of the submitted methods are summarized in the video due to the page limit.

5 Results and Findings

Table 3 reports the official leaderboard of the participants' methods. It is observed that Team Intro to AI team 8 slightly outperforms the competition Team Badminseok, while Team YHY demonstrates the best performance in terms of area predictions. 11 teams perform better than ShuttleNet; however, we notice that these methods are built on top of ShuttleNet, which highlights the flexibility of ShuttleNet and the potential for improvements in certain aspects, e.g., hyperparameters and activation functions. In addition, the improvements made by the participating teams primarily focus on the shot type prediction (2.1777→1.7892); however, the performance of area coordinates only marginally surpasses ShuttleNet (0.6997 → 0.6797), with most teams being inferior to the baseline. This underscores the challenge of effectively integrating two predictions, which may be a potential future direction for exploration.

6 Conclusion

In this paper, we propose ShuttleSet22, an extended dataset from ShuttleSet with stroke-level badminton singles records. ShuttleSet22 consists of fine-grained metadata to reinforce researchers to explore various aspects and derive insightful findings. Exploratory data analysis is conducted to elucidate the fundamental compositions of the dataset, aiming to bridge the gap between researchers outside the badminton domain. To foster researchers to incorporate advanced techniques into badminton analytics, we introduce Track 2, a challenge within CoachAI Badminton Challenge 2023, focused on enhancing the effectiveness of stroke forecasting. In addition, we establish the state-of-the-art baseline for the task. We summarize and discuss the methods and insights from participants to provide potential avenues for future research improvement.