A Human-in-the-Loop Tool for Annotating Passive Acoustic Monitoring Datasets

1 Introduction

Machine learning (ML) with deep neural networks has achieved excellent performance in many tasks. Yet, the impact of ML on several domains is limited by technical entry barriers, as well as by lack of domain-specific annotated data for supervised learning. Motivated by the quest to improve efficiency of passive acoustic monitoring (PAM) of animal biodiversity, we are developing a graphical interactive ML tool for detection and annotation of events in PAM datasets.

PAM is an increasingly popular method for continuous, reproducible, scalable, and cost-effective monitoring of animal wildlife [Sugai et al., 2018]. While available low-cost recording devices have allowed large-scale data collection [Hill et al., 2019], processing this data is a bottleneck.

Due to the low quality of automatically generated annotations for PAM datasets, annotation is usually done manually: domain experts listen to each audio file, annotating events by manually selecting time segments on a graphical representation of the sound (e.g. amplitude envelope or spectrogram) [Audacity Team, 1999; Tkachenko et al., 2020; Perry et al., 2021]. This approach is laborious and incompatible with the large volume of data generated by PAM. Seadash proposes a graphical implementation of data programmingsimultaneous whole-dataset annotation with a set of userdefined heuristics [Ratner et al., 2016]-but hasn't been evaluated on real life datasets [Gouvˆea et al., 2022]. DetEdit is a ML-free tool that allows simultaneous detection of bouts of events through a configurable signal processing pipeline that includes a GUI for accepting/rejecting detections; it runs on a proprietary platform, and has only been evaluated on odontocete echolocation click datasets [Solsona-Berga et al., 2020]. scikit-maad is a tool for large scale PAM data analysis by spectrogram segmentation and clustering [Ulloa et al., 2021]; as a command line tool, it lacks interactivity.

We present an interactive ML-based tool for annotating PAM datasets 1 . Implemented features are derived from audio annotation tools and domain expert experience. Our approach addresses three shortcomings of existing tools by allowing multiple events to be annotated simultaneously, using additional annotations to continuously speed up the process rather than following a linear annotation speed, and using clickable labels rather than error-prone manual input. The underlying deep generative model (DGM) [Rezende et al., 2014] improves the machine predictions using the human-in-the-loop concept [Monarch, 2021] and distorts the latent space to represent events as outliers. Interactive tools using the latent space of ML systems can facilitate data interaction [Prange and Sonntag, 2021]. While many datasets (e.g. Xeno-canto 2 ) are annotated weakly (i.e. on file level) and current tools create strong (i.e. timestamp level) labels [Perry et al., 2021; Grover et al., 2020], we use time- and frequency-aligned labels visualised as bounding boxes, allowing noise reduction and time-overlapping annotation (see figure 1, spectrogram).

1 https://www.youtube.com/watch?v=VOfohkiWevU

2 https://xeno-canto.org/

2 SystemDescription

User Interface. We designed a Dash 3 interface composed of three parts, namely the file selection bar, the spectrogram interaction row and the state-space interaction row (see figure 1). The file selection bar allows the user to select an audio file for annotation from a list. The spectrogram interaction row displays annotation tools over and alongside the spectrogram of the selected file. Spectrograms are main elements of current audio annotation tools and lend themselves to an intuitive presentation of raw data. Hovering over the spectrogram reveals a toolbox that allows the user to zoom in and out, as well as create and edit time and frequency aligned bounding boxes to annotate regions of interest (ROIs). Selected ROIs can be played as audio. A button below the spectrogram allows quick annotation by suggesting bounding boxes and associated labels. Assigning, changing, and saving labels is possible through selection elements displayed alongside the spectrogram. More processed representations of the selected audio file are shown in the state-space interaction row, with each dot representing a colour-coded second in both figures. The left figure shows representations of the pre-trained unsupervised learning model, the right figure the fine-tuned model that also processes (generated) annotations. Hovering over the figures reveals a toolbox that allows users to zoom in and select data points, which are highlighted in the two figures and used to create a table of associated times on the right. A button below uses these times to create bounding boxes in the spectrogram. Continuous fine-tuning of the model using all (generated) annotations is possible by selecting a number of epochs and the corresponding button. Another button resets the model to the pre-trained state.

3 https://dash.plotly.com/

Workflow. Figure 2 shows the workflow of our system. Starting from an incompletely annotated dataset, the user loads the data used to train both models. The user selects a file for annotation in the file selection bar, which is displayed as a spectrogram and state-space representations of two different models. The spectrogram contains existing annotations. Adding annotations is done by one of three ways: The most intuitive way for experts is to draw bounding boxes directly on the spectrogram. Secondly, the user can retrieve a bounding box predicted by the fine-tuned system. And thirdly, the user can select one or more points in the state-space representation and create bounding boxes around the selected times. Our fine-tuned model is designed to represent events as out-

save manually created annotations

liers, making it easy for the user to create accurate bounding boxes around any time-aligned regions of interest. Regardless of how the bounding boxes are created, the user can assign labels to them and move, scale and delete them in the spectrogram. Saving intermediate or final results is possible via a button and leads to an improvement of the dataset. This improvement can be used by selecting a number of epochs and re-training the model, resulting in more accurate bounding box suggestions as well as better separation in the state-space representation. The improvement of our model enables faster annotation of the next audio file.

Architecture of DGM. The requirements for the deep generative model include learning relevant data structures without annotations, mapping these structures into a 2D latent space, and a way to helpfully customise the latent space for the user based on the added annotations. Our derived model architecture is inspired by [Paige et al., 2017] and extends a variational autoencoder (VAE) [Kingma and Welling, 2014] with a classification head. The input data X is processed by the encoder and mapped to the 2D-latent variable Z, which is presented to the user as a state-space and represents the input to the decoder and classifier. The decoder computes the reconstruction ˜X. The classifier is implemented as a multilayer perceptron (MLP) and processes only annotated data by computing predicted labels ˜ Y from Z for all files with a label Y , grouping in Z data points of the same category. The VAE and MLP are jointly optimized by minimizing the loss function

where the first two terms are as in [Kingma and Welling, 2014], and H is the cross entropy between Y and ˜ Y . Efficient storage of bounding boxes is implemented using tidy data tables [Wickham, 2014]. Data pre-processing includes the calculation of the spectrogram and the subdivision of the audio files into second-long units. The DGM (referred to as finetuned model) implemented in Tensorflow 4 , in the absence of labels, is identical to a VAE (referred to as pre-trained model) displayed for comparison purposes. While the autoencoder is trained on the entire dataset, the classification head only processes regions of bounding boxes, which is why conspicuous but uninteresting events (e.g. artefacts, geophony) can easily be ignored by the DGM.

4 https://www.tensorflow.org/

3 Preliminary Evaluation

We ran preliminary evaluations of bounding box prediction and the usability of state-space representations. We created a dataset of 50 one-minute audio files composed of a PAM background (recorded in the Central Catchment Nature Reserve, Singapore) and foreground events from the UrbanSound8k dataset [Salamon et al., 2014] inserted at random times (Poisson distributed, average of 4 per file). The VAE was trained on the entire dataset (pre-training); 30 files were used for fine-tuning (100 epochs), and the remaining 20 files for evaluation.

Bounding box prediction was evaluated by predicting three bounding boxes per file. Each bounding box capturing an event was considered correct. The prediction accuracy of the fine-tuned model is 79.9%.

To obtain initial empirical results, we had a user create bounding boxes by selecting all elements in the statespace representations that were considered outliers of the pretrained and fine-tuned model. For evaluation, we categorised selected items that contain events (true positive), selected items that do not contain events (false positive) and unselected items that contain events (false negative). To treat all events and predictions equally, we used the respective sums of all 20 evaluated files for the F-score calculation. The Fscore of the pre-trained model is 77.0%, the F-score of the fine-tuned model is 94.2%.

4 Conclusion and FutureWork

We propose an interactive, human-in-the-loop tool for MLassisted annotation of PAM datasets. By modifying the latent space of a VAE through an added classifier head, we generate an actionable, low-dimensional representation of the input data that can improve efficiency of event detection and classification by the user. Future work includes making our tool applicable to real-world problems [Gouvˆea et al., 2023]. The webapp will be integrated with the users's PAM database and served remotely. To improve our tool in terms of implemented features (e.g. providing frequency units in the statespace representation, enabling playback of selected spectrogram regions, implementing existing libraries such as [Ulloa et al., 2021], real-time update of the system) and interface design, we plan to follow a human-centred AI approach. Using design science research methods [Peffers et al., 2008] we plan to conduct a user study with domain experts.

Ethical Statement

There are no ethical issues.

Acknowledgments

We thank Simone Dena and the Fonoteca Neotropical Jacques Vielliard (FNJV) for providing passive acoustic monitoring data. We thank Patr´ıcia P. Serafini and Ivan Campos for sharing their experience in annotating passive acoustic monitoring datasets.