CVAT-BWV: A Web-Based Video Annotation Platform for Police Body-Worn Video

1 Introduction

High-quality policing is an essential government service, yet it often unevenly impacts different communities [California Department of Justice, 2024]. In the US, police conduct over 20 million traffic stops annually, where the officers' communication, including their tone of voice and level of respect, shapes public perceptions of, and trust in, the police [Camp et al., 2021]. The vast majority of these traffic stops are captured on body-worn video (BWV). Unfortunately, even large police departments currently have limited ability to analyze this data. Instead of transforming policing, BWV data often serves merely as an ad hoc crisis management tool, with footage rarely reviewed unless a complaint is filed or a tragic event occurs.

Machine learning (ML) tools have the potential to enable police departments and those tasked with their oversight to analyze BWV footage at scale, thus facilitating transparency and accountability. Relatively small-scale studies of BWV footage have already revealed significant racial disparities in officer-civilian interactions [Voigt et al., 2017; Rho et al., 2023; Prabhakaran et al., 2018], highlighting the potential of BWV footage if analyzed effectively. However, developing ML models in the policing domain requires extensive human annotation of BWV recordings by a diverse population of annotators, not least because the perception of effective communication can differ widely between individuals and communities [Graham et al., 2024]. These human annotations can help distinguish between speakers in overlapping dialog, track the progression of conversations, identify the (de)escalation of conflict, analyze communication styles from gestures to tone of voice, and also capture potential differences in viewpoints of groundtruths.

Several annotation tools have been developed to enhance the annotation process for multimodal ML tasks (e.g., Labelbox, Prodigy); however, none offer the comprehensive functionalities required for annotating BWV data. We have based our framework on Computer Vision Annotation Tool (CVAT) [Sekachev et al., 2020], as it is an open-source annotation software that natively supports object detection and image classification. Moreover, it offers the flexibility of local hosting and the customizability inherent in open-source programs. Other existing tools, such as Labelbox [Labelbox, 2024] and Prodigy [Montani and Honnibal, 2024], were not suitable for our task; they either operate as Software as a Service platforms, which do not meet the security criteria for sensitive data like BWV, are incapable of handling all modalities (text, audio, and video), or are not sufficiently open-source and customizable to incorporate ML models for assisted annotation and enable modification of the user interface.

We introduce CVAT-BWV 1 , an open-source multimodal interactive annotation platform for BWVs with the following features: 1) Secure and local processing of data, due to the sensitive nature of BWV footage; 2) Integration of AI for assisted annotation in various tasks, including object detection, automatic speech recognition (ASR), speaker diarization, and face recognition; and 3) Support for multimodal, and multiperspective, annotations.

1 https://www.youtube.com/watch?v=LyfLsG8OQug

2 PlatformDescription

CVAT-BWV 2 is an open-source, web-based application that can be hosted on a local network, eliminating the need for an internet connection and thereby ensuring data privacy. Additionally, it incorporates user management to control access to annotator accounts and their assigned tasks. As a multimodal platform, it supports annotations for transcripts, auditory content, and visual entities, all of which are enhanced by AI-assisted annotation tools.

2.1 AI-Assisted Annotation Tools

Incorporating Human-in-the-Loop components in annotation tasks can enhance data quality and user experience while reducing the annotators' workload [van der Wal et al., 2021; Marchesoni-Acland and Facciolo, 2023; Weber and Plank, 2023]. CVAT-BWT employs several AI models to assist with various annotation tasks:

ASRand Speaker Diarization. The platform leverages the audio channel in BWV footage to transcribe and segment the audio stream based on speaker identity. This is achieved through the WhisperX model [Bain et al., 2023], which provides accurate word-level timestamps.

Object Detection. For automated bounding box generation around people and vehicles, the platform integrates YOLOv8, a state-of-the-art object detection model [Jocher et al., 2023].

2 https://github.com/USC-CSSL/CVAT-BWV

Face Recognition. To avoid duplicate bounding boxes for the same individual, newly created bounding boxes of people are fed into the LightFace face recognition framework [Serengil and Ozpinar, 2020] to be matched with previously created bounding boxes. We use the RetinaFace [Deng et al., 2020] together with the FaceNet face recognition model [Schroff et al., 2015] from the framework.

2.2 User Interface

The main annotation page of the tool comprises seven principal components (see Figure 1). (1) Media Playback Header offers controls for playing the video at regular speed or frame by frame. Annotators can navigate to a specific frame using the frame selector, control buttons, or the playback slider. Additionally, this header includes options for accessing the menu, saving progress, and performing undo or redo actions. BWV footage is displayed in the (2) Video Display Area, where annotators can create and modify visual annotation objects (bounding boxes). This canvas also presents the bounding boxes overlaid on each frame of the video, complete with their assigned labels and identifiers. (3) Interactive Control Toolbar provides controls for annotators to drag, zoom, and rotate the video canvas. It also provides tools to manually add bounding boxes by drawing over the Video Display Area or through an automated object detection tool. (4) Audio Waveform Display Slider illustrates the audio waveform's amplitude variations over time in the video. It also displays transcript segments with color-coded handles-termed "transcript timing handles." Annotators can create a new transcript or adjust the start and end times of an existing transcript by dragging these handles along the waveform. (5) Off-screen Audio Tagging Bar is used when there is an audible utterance in the video whose speaker is not visible. In such instances, annotators can create auditory annotation objects, known as audio tags. Using the "audio timing handles" on the waveform track, annotators can adjust the start and end times of the utterance and assign labels to the audio tags. (6) Annotation Objects Sidebar lists bounding boxes and audio tags created by annotators for the current frame or throughout the entire video. Each entry on this list corresponds to an annotation object and includes a button for navigating to the precise frame of its creation. The assigned label or identifier for each entry can be modified, or the entry can be deleted if necessary. (7) Interactive Transcript Sidebar displays the complete transcript of the video, highlighting the currently spoken utterance. Initially, this sidebar is populated with the output from the ASR module, providing an auto-generated transcript and the results of speaker diarization, each marked with a unique colorful avatar. Annotators can then correct any errors. Moreover, inaccuracies in speaker diarization can be rectified by reassigning colors to utterances.

2.3 Workflow

Task Creation and Assignment. In CVAT-BWV, annotation begins with the task creation step, where raw BWV footage is uploaded, label categories and subcategories can be modified or created, and tasks can be assigned to annotators. While the video is uploading, the ASR and speaker diarization module extracts transcripts and identifies speakers in the background.

Transcript Annotation. After task assignment, annotators can access the task. The Interactive Transcript Sidebar displays utterances from detected speakers, each represented by a uniquely colored avatar. Additionally, transcript timing handles appear in the Audio Waveform Display Slider, colorcoded to match the avatars of the speakers.

Creation of Bounding Boxes and Audio Tags. On each video frame, annotators have the option to pause and create visual or auditory annotation objects. Bounding boxes can be manually drawn over a frame or automatically generated using the object detection module for identifying people and vehicles. Audio tags are created using the Off-screen Audio Tagging Bar and the audio timing handles on the waveform tracks. Annotators then assign one of the predefined labels to each annotation object created during the task creation step. If a "Person" tag is assigned to a bounding box, the face recognition module processes it to check for duplicates among previously created face bounding boxes, alerting the annotator with a warning in case of a previously tagged person. However, the decision to dismiss the warning is left to the annotator, given the face recognition module's potential inaccuracies. The de-duplication process is repeated upon task completion to ensure no multiple bounding boxes have been created for a single individual.

LinkingTranscripts toAnnotationObjects. At any point, it is possible to link a uniquely colored avatar from the transcripts sidebar to one of the visual or auditory annotation objects, with this assignment automatically applied to all avatars of the same color.

Annotation Object Questionnaires. After the creation of annotation objects, annotators can be prompted to complete questionnaires specific to each created annotation object, whether a bounding box or an audio tag, based on their assigned label.

3 Conclusion

Many of the most consequential government-civilian interactions in our democracy, including those involving police, are captured on body-worn cameras or similar devices. The data from these devices hold enormous potential to facilitate transparency and accountability efforts, but only if effective ML tools can be developed to analyze this footage. Effective tools to analyze these complex social interactions begin with extensive human annotation.

The CVAT-BWV platform introduced here provides four core features not available in current annotation tools: 1) The ability to operate in secure data environments suitable for sensitive or classified data; 2) Flexible customization and AI integration for automatic labeling of domain-relevant audio and video features; and 3) Support for multimodal and multiperspective annotations. Our tool is released under an MIT/X license, and we hope its widespread adoption will facilitate rapid progress in developing the tools necessary to empower the use of BWVs as a tool for learning and oversight.

Ethical Statement

The nature of the BWV footage, often capturing highly sensitive and potentially violent interactions, necessitates careful consideration of the annotators' well-being. Acknowledging the emotional and psychological toll that extended exposure to such content can have, it is crucial to monitor for signs of secondary trauma or post-traumatic stress disorder among those tasked with annotation. Implementing measures to support mental health and mitigate adverse effects is not only ethical but essential for responsibly handling the annotation of BWV data.

Acknowledgments

We thank our anonymous reviewers for their valuable feedback. This work was funded by the Microsoft Justice Reform Initiative, National Science Foundation Civic Program, Arnold Ventures, Google Award for Inclusion Research, and USC Zumberge Interdisciplinary Grant. The views expressed are those of the authors and do not necessarily represent the positions of the funders.

Contribution Statement

Authors Preni Golazizian, Rajat Hebbar, and Jackson Trager contributed equally to this work.