Near Real-Time Detection of Poachers from Drones in AirSim

1 SPOT

SPOT includes two main parts: (i) offline training and (ii) online detection. Fig. 1 shows the steps of both, but we now introduce online detection in detail, with an emphasis on the processing techniques we use to improve efficiency and provide detections in near real time in our demonstration.

1.1 Running

We treat each frame of the video stream as an image and pass it to Faster RCNN [Ren et al., 2015]. The trained model computes regions and classes associated with each region.

1.2 Architecture and Efficiency

Faster RCNN runs at 5 frames per second (fps) on a K40 GPU. Because the live video is 25 fps, efficiency is vital to similar timing with limited computational power in the field. Note that GPUs have recently become available in the field. However, in the case that they are not available, we also provide a solution taking advantage of Microsoft Azure. To communicate between the client and Azure, Tensorflow Serving, a framework for operationalizing trained Tensorflow computation graphs, provides a way to evaluate Faster RCNN with less overhead and efficient communication. We also use azure-engine to create a cluster of NC-6 series GPU VMs managed with Kubernetes, a fault tolerant load balancer for scalable cloud-based services. We deploy on a GPU-enabled docker image with Tensorflow Serving, and add tools for convenient and automatic re-deployment of models hosted on Azure Blob Storage. In the field, SPOT is deployed with the option to toggle between remote and local computation. For the purposes of the demonstration, we will make use of the local compute option on an Azure virtual machine.

2 RelatedWork and Design Choices

We arrive at the current framework of SPOT after several rounds of trials and errors with relevant traditional computer vision techniques, including automatic thresholding [Otsu, 1979], motion algorithms [Lucas et al., 1981], and shape detection algorithms, such as EyeSpy, the application Air Shepherd uses in current practice [Hannaford, 2017]. All struggle due to the difficulty of thermal infrared images (Fig. 2). There has also been some work on automatic wildlife detection in UAV videos, but they either rely on RGB images in high resolution [Olivares-Mendez et al., 2015] or do not consider real-time detection [van Gemert et al., 2014]. Given these limitations and the great strides in object detection using convolutional neural networks, we turned to Faster RCNN.

3 Evaluation

We first evaluate SPOT using six historical videos, consisting of 15,403 frames in total, as well as the field test video with poachers (Fig. 2). We compare to EyeSpy, the application that is used in current practice. It requires users to tune 14 parameters, which is difficult for a novice, whereas ours can be used without tuning. We therefore consider EyeSpy as used by a novice by utilizing default values for all parameters. We use an IoU threshold of 0.5 and a low confidence threshold for all calculations, and we report the precision and recall. We compare the performance of SPOT and EyeSpy on videos containing animals and poachers with labels of small, medium, or large average sizes in Tables 1 and 2. We perform better than the novice in both precision and recall for medium- and large-sized poachers and animals, and in the field test video. Because we perform better than EyeSpy, we thus reduce significant burden. For small poachers, which is a challenging task for object detection in general, both tools perform poorly, with EyeSpy being able to identify a small number of poachers correctly. To improve performance for small objects in the future, we expect pooling the results of video frames and incorporating motion will be beneficial.

Next, we evaluate efficiency by comparing the field CPU and GPU laptops, and Azure system performance. We conduct the Azure tests in two different networking environments: 533.20 Mbps upload and 812.14 Mbps download, which we call "fast", and 5.33 Mbps upload and 5.29 Mbps download, which we call "slow". We repeat the experiment for several images and show the final time per image in Table 3. The results show that both Azure and the GPU laptop perform detection much faster than the CPU laptop.

4 Demonstration

In order to demonstrate SPOT, we utilize AirSim [Shah et al., 2017]. AirSim is a simulator for drones and cars built on Unreal Engine to allow artificial intelligence researchers to experiment with deep learning, computer vision, and reinforcement learning algorithms for autonomous vehicles. For a live demonstration of SPOT in an African savanna, we built an African savanna environment in Unreal Engine. We also modeled thermal infrared frames using a simple physicsbased model and fine-tuned Faster RCNN with the simulated infrared frames to improve detection results [Bondi et al., 2018a]. Results shown in Tables 1 and 2 use real data only.

A sample video can be found on Google Drive. The video provides the scene and corresponding thermal infrared images. This is an example of navigation in AirSim with a Python API, along with the near real-time detections of SPOT. Demonstration users will be able to interact with AirSim by flying around the environment.

In conclusion, SPOT is a novel decision aid to assist the UAV crew in detecting poachers and animals in thermal infrared videos captured aboard drones. It performs better than EyeSpy, both on historical data and in the field, provides detections in near real time, and does not require tuning by the user during run time. The promising results from the field test have led to a plan for larger-scale deployment in a national park in southern Africa. To demonstrate SPOT, we utilize an African savanna environment in AirSim that allows users to interactively fly through a savanna, while also running SPOT for live, near real-time detection.

Acknowledgments

This was supported by Microsoft AI for Earth, the National Science Foundation (CCF-1522054), and UCAR N00173-162-C903, with the primary sponsor being the Naval Research Laboratory (Z17-19598). It was partially supported by the Viterbi School of Engineering Ph.D. Merit Top-Off Fellowship.