GraspSnooker: Automatic Chinese Commentary Generation for Snooker Videos

1 Introduction

Snooker is an attractive sport which needs not only physical skills but also a lot of strategies. Novices of snooker may have difficulty in watching raw game videos without commentaries. Therefore automatically generating commentaries for snooker game videos can help them learn from professional players and also add fun to game watching.

Recently, several game commentary generation works have been developed. Yu et al. [2018] proposed a fine-grained NBA video narrative framework, and Jhamtani et al. [2018] presented a move-by-move commentary generation method for chess games. However, there is still no such work on snooker games. Moreover, deep analysis of sport games such as strategies was ignored in most previous approaches. Although some robotic billiards players have been introduced [Smith, 2006; Archibald and Shoham, 2009], their systems were designed under physical environment or simulation environment, not applicable for analyzing strategies from video and generating commentaries.

In this paper, we present a pipeline based system, GraspSnooker, to generate Chinese commentaries for snooker game videos. As far as we know, it is the first snooker video commentary system. For simplification, we generate commentaries for each single shot video which starts with a stationary table without moving balls and ends with a stationary table after one stroking. We collect 10.8k single shot videos and annotate 15.7k corresponding commentaries for training.

The system consists of three components: a) a video analyzer, b) a strategy predictor, and c) a text generator. The video analyzer processes videos, the strategy predictor predicts the target ball and pocket, and the text generator produces commentaries. As snooker commentaries are flexible, we classify them into three categories: score commentary, prediction commentary and track commentary. Specifically, score commentaries describe the score status of players, prediction commentaries show the narrators' guesses about the next shot, and track commentaries explain what happens during this shot. Some examples are shown in Table 1.

∗Contact Author

2 Demonstration Outline

Users can upload a single shot video to GraspSnooker, and the system will process the video and return the commentaries. On the upload page, users need to provide the information of whether it is required to hit a red ball or a coloured one in this shot. The result page shows the uploaded video with detected table and balls and its perspective mapping to a standard ratio table. The predicted target ball and pocket are marked by twinkling arrows. Commentaries generated by two different methods are shown below the videos. English commentaries generated from templates are also provided to assist in understanding. Table 1 gives some commentary examples. GraspSnooker also provides a few preprocessed video clips which can be used to generate commentaries.

3 SystemArchitecture

The system architecture of GraspSnooker is presented in Figure 1. The system mainly consists of three modules: 1) a video analyzer, which detects the table and balls from video screens, and tracks balls to extract important movement information; 2) a strategy predictor, which predicts the target ball and the target pocket; 3) a text generator, which generates commentaries based on the results generated from the video analyzer and the strategy predictor. The details of these three components are described in the following three subsections.

3.1 Video Analyzer

For better video understanding, it is crucial to detect the table and balls in the video (locating) and capture the traces of the moving balls (tracking).

For locating, front view frames showing the full table at a fixed angle are first extracted. Then the table is separated by green borders. And balls are detected by template matching. Once the table and balls are located, locations are mapped to a standard ratio table by perspective transformation.

Table 1: Comparison of texts generated by template-based and keyword-based methods. Blue marks differences. Red marks mistakes.

For tracking, moving balls are detected using background subtracting. And the positions of each ball are reduced to a simple polyline. For convenience, spins are out of consideration. For extracted traces, some crucial information such as which balls are bumped and which cushions are hit, are extracted. These key features will be used to generate commentaries described in Section 3.3. We implement the video analyzer using the python package of OpenCV 1 .

3.2 Strategy Predictor

The role of the strategy predictor is to predict which ball to hit and which pocket to target. An adaptation of graph neural network [Li et al., 2015] is employed to learn from data.

Specifically, each ball except the cue ball on the start table is considered as a vertex in a graph. The initial feature of a vertex is a 28-dim vector, including the color, position and some geometric features inspired by [Smith, 2006]. And for a given ball, its nearest N balls are defined as its neighbors. N is set to 8 here. The edge feature between a ball and one of its neighbors includes two values: the distance between the ball and a neighbor, and the cosine value of target ball-cue ballobject ball angle. Then a similar propagation model as [Li et al., 2015] with 2 layers can be used for balls' feature propagation. Next, for the output layer, ball and pocket predictions are learned simultaneously. And the output of the ball prediction is used as a soft attention that decides which balls are relevant to the pocket prediction.

The prediction model is trained on 2k training instances and 200 validation instances. Tested on 200 instances, the accuracy of target ball prediction is 62.25%, of target pocket prediction is 77.04%, and of both is 56.12%. The prediction model is implemented based on PyTorch Geometric 2 library .

1 https://pypi.org/project/opencv-python/

2 https://rusty1s.github.io/pytorch geometric/build/html/index. html#

3.3 Text Generator

In our demonstration, two solutions are provided to generate commentaries: template-based generation and keywordbased generation. The template-based method generates sentences from human-written templates, while the keywordbased method extends given keywords of templates to sentences by CGMH (Constrained sentence Generation by Metropolis-Hastings sampling) [Miao et al., 2018], aiming to increase the language diversity. CGMH is an unsupervised method applying Metropolis-Hastings sampling [Metropolis et al., 1953] to generate sentences under constraints. Readers may refer to [Miao et al., 2018] for details of CGMH.

Examples of some commentaries generated by the two methods are illustrated in Table 1. Compared to templatebased generated commentaries, keyword-based generated commentaries are more flexible. Blue texts show their differences. However, as CGMH only generates sentences according to language models, ignoring the context of the events, so that it usually loses some coherence. In addition, sometimes words that are contrary to the video information are introduced, which are marked as red in Table 1.

Two language models of forward and backward direction are trained on all 15.7k collected snooker commentaries, which get perplexities of 35.87 and 36.04 respectively.

4 Conclusion

In this work, we present GraspSnooker, the first application on snooker commentary generation. It combines video analysis and strategy prediction with commentary generation. Long video commentary generation will be explored next.

Acknowledgments

This work was funded by the National Key R&D Program of China (No. 2017YFB1002801), the National Natural Science Foundation of China (61772132) and the Natural Science Foundation of Jiangsu Province of China (BK20161430).