Demo: Enhancing Wildlife Acoustic Data Annotation Efficiency through Transfer and Active Learning

1 Introduction

Biodiversity loss is among the most pressing issues of our days [Cardinale et al., 2012]. Drivers of the negative change have been accelerating, and meeting internationally agreed conservation targets will require transformative change [IPBES, 2019]. While machine learning methods have been increasingly brought to bear to support wildlife management, the tools available to those on the ecological front lines still lag the state-of-the-art in artificial intelligence research [Tuia et al., 2022; Gouvˆea et al., 2023]. Passive acoustic monitoring (PAM) has emerged as a powerful technology for wildlife monitoring, allowing ecologists to gather extensive data on wildlife with minimal disturbance of habitats [Sugai et al., 2019; Sugai and Llusia, 2019]. PAM systems can be used to continuously record sounds from various biomes, offering valuable insights into animal behaviour, species richness, and ecosystem health, with applications in ecosystem management, rapid biodiversity assessments [Sueur et al., 2008], and basic research [Ross et al., 2023].

However, effectively utilising this vast amount of data still poses significant management and analysis challenges. Sound event detection in particular (e.g., species identification) is limited by the need for annotated data to train supervised machine learning models. PAM data annotation is usually done manually, in a laborious and time-consuming process: domain experts listen to each audio file, annotating events by manually selecting time segments on a graphical representation of the sound (e.g., amplitude envelope or spectrogram) [Tkachenko et al., 2020; Perry et al., 2021; Ca˜nas et al., 2023]. This approach is laborious and incompatible with the large volume of data generated by PAM, and current research focuses on automating the annotation process. Seadash introduces a graphical implementation of data programming, but lacks evaluation on real life datasets

[Gouvˆea et al., 2022]. DetEdit allows simultaneous detection of bouts of events through a configurable signal processing pipeline that includes a GUI for accepting/rejecting detections; it runs on a proprietary platform, and has only been evaluated on odontocete echolocation click datasets [SolsonaBerga et al., 2020]. Scikit-maad [Ulloa et al., 2021] and BamScape [Michaud et al., 2022] are tools for large scale PAM data analysis by spectrogram segmentation and clustering; as command line tools, they lack interactivity.

A promising, under-explored direction of research on interactive PAM annotation is the leveraging of small amounts of available labels (e.g., at early stages of annotation of a novel dataset) to improve the efficiency of the annotation process. We have previously shown that early annotations can be used by a semi-supervised dimensionality reduction method to generate actionable scatter-plots, speeding up the annotation process by facilitating grouping of similar samples as well as identification of outliers [Kath et al., 2023].

As an interdisciplinary team of ecologists, ecosystems managers, and machine learning researchers, in the present work we introduce a PAM annotation tool that builds on our previous work by leveraging active learning [Kadir et al., 2023] in an embedding space learned in a related domain.

2 SystemDescription

Machine Learning Backend. It has been shown that BirdNet, a neural model trained primarily on focal recordings of songbirds [Kahl et al., 2021], learns embeddings that are useful for sound event detection on PAM data of different species groups [Ghani et al., 2023; Kath et al., 2024a]. Following [Kath et al., 2024a], our system splits sound inputs into 3s segments and embeds them into one of the three penultimate layers of BirdNet. A suite of sampling algorithms for multi-label active learning are subsequently used to select segments for annotation. The suite of methods includes uncertainty-based (average entropy, least-confidence, or ratio, as well as max ratio) and diversity-based methods (see [Kath et al., 2024a] for details.) In the tool introduced here, the user makes use of a graphical interface to control the choice of embedding layer and active learning sampling methods, as well as for evaluating and annotating each sample.

User Interface. We designed a multi-tab Dash 1 web application composed of three interfaces, namely the Data Source tab, the Settings tab and the Annotation tab 2 (see fig. 2). The Data Source tab controls whether the input data is read from the local computer or a remote location-in the current version we considered practical to make use of a commercial cloud provider (Google Drive). The settings tab controls the choice of transfer learning embedding model and layer, and the composition of active learning sampling methods. In addition, data preprocessing is triggered from this tab: each file in the selected dataset is split into 3-second audio segments (snippets), and their corresponding embeddings are computed using the selected embedding model and layer. Lastly, the Annotation tab presents the selected snippets along with resources for evaluation and label assignment. The layout of this tab is composed of three columns: snippet selection, audio representation, and annotation (fig. 2). The snippet selection area lists the sound snippets queued for annotation, along with the sampling method used to select them. In addition, it is also possible to select samples manually. The Audio Representation area displays the currently selected sample as a spectrogram. Selecting a time-frequency box on the spectrogram allows listening to the corresponding band-pass filtered segment. The Annotation area allows assigning labels (e.g., species id) to the selected audio snippet by ticking boxes on the list of label classes. New classes can be entered through a text input field. The active learning model can be used for inference as well: predicted labels can be accepted by clicking the Model→Manual button. A text display on the bottomright corner indicates settings chosen on the Settings tab.

1 https://dash.plotly.com/

2 https://youtu.be/mAVm79-UmlA

Workflow. While fig. 1 illustrates the data-centric workflow, here we outline the workflow from the user perspective. When the web application is accessed, only the first tab (data source) is available. Upon selection of a data location, the system validates it (either by checking the local path or an authorisation code provided by Google Drive.) Next, on the Settings tab the user can select a dataset folder, as well as an embedding model and layer, and an active learning strategy. The active learning strategy is composed of controllable fractions of random sampling (as in ϵ-greedy policies), a diversitybased method, and a choice of uncertainty-based method (see [Kath et al., 2024a] for details.) This composite strategy is used to select the audio samples (snippets) to be presented to the user. Next, the user triggers the pre-processing of the data, a process cached separately for different transfer-learning settings values. While these settings remain modifiable at any time, the actual annotation of data takes place in the Annotation tab, which becomes active after pre-processing. The user determines the species present in the selected sample by examining the spectrogram and listening to the audio. By selecting a time-frequency region in the spectrogram, the audio is band-pass filtered, in some cases allowing the user to identify species calls even when obscured by loud background noise (provided they don't overlap in time and/or frequency). The user selects the present species in the Manual column of the annotation table. If the active learning model provides satisfactory predictions the user can simply accept them by pressing the Model→Manual button. Active learning gains deteriorate when optimising for too many classes at the same time [Kath et al., 2024b]; the tool therefore allows the user to mark to which label classes should active learning attend by ticking the boxes on the AL column of the annotation pane (fig. 2). After annotating the sample snippet, the user saves the annotations via the Submit button. If the user is unsure of the species present, they can skip the annotation for later review using the Skip button. Pressing the Submit or Skip button loads the next sample for annotation. If the user wishes to select a sample manually, e.g. to inspect snippets based on metadata, or to revisit skipped snippets, they can do so using the Snippet Selection area.

3 Preliminary Evaluation

Preliminary evaluations are being conducted using a real life dataset collected at Fernando de Noronha, a designated protected area recognised by UNESCO as a World Natural Heritage site. The application was deployed on HuggingFace where a domain expert (who co-authors this study) used the tool for identifying and labelling vocalisations by oceanic birds. The domain expert described the workflow as particularly intuitive, although opportunities for improvement were identified. In particular, when there is uncertainty about the species identified, it is common practice to listen to example files within the Xeno-Canto database 3 . The integration of example links within the tool is suggested to streamline and improve the workflow.

In addition, every interaction with GUI elements is being logged for a subsequent usability study. Figure 3 shows raw results from an example annotation session. A previous study with this same dataset and user [L¨uers et al., 2024] showed that, using the traditional tools, the domain expert needs 648 ± 301 seconds to annotate each 60-seconds long input file, a factor of 10.8 ± 5.0 relative to real time (mean ± standard deviation). In the annotation session depicted in fig. 3, the median annotation time for a 3-second audio file was 11 seconds with our tool, a factor of less than 4. While anecdotal, this represents a promising first piece of evidence that our tool can have a significant impact on the manageability of large PAM datasets.

4 Conclusion and FutureWork

We present an annotation tool designed for passive acoustic monitoring datasets that uses a combination of transfer learning and active learning strategies. This approach aims to generate meaningful representations of the data and present the most informative samples to the user for annotation. The tool includes an interactive user interface designed to be applicable to real-world problems. Preliminary evaluation shows promising results, with a remarkable acceleration of the conventional annotation process by a factor of 2-4, accompanied by an intuitive user experience. Future work includes conducting a more extensive user study and incorporating requested features. One notable enhancement is the integration of sound samples from sources such as Xeno Canto.

3 https://xeno-canto.org/

Acknowledgements

This research is part of the Computational Sustainability & Technology project area 4 , and has been supported by the Ministry for Science and Culture of Lower Saxony (MWK), the Endowed Chair of Applied Artificial Intelligence, Oldenburg University, and DFKI.