Learning-Based Tracking-before-Detect for RF-Based Unconstrained Indoor Human Tracking

1 Introduction

Passive human tracking which predicts multiple target trajectories without requiring body-worn sensors, is one of the most fundamental yet challenging topics in wireless sensing [Thormann et al., 2018; Zhang et al., 2019a; Zhang et al., 2019b; He et al., 2020]. Previous efforts employ thresholding on signal measurement to maintain high tracking accuracy. However, hard-thresholding presents severe performance degradation encountering complex indoor scenarios. Higher threshold leads to increased number of false alarms, referring to false positive tracks, while lower threshold leads to the loss of true positives. To resolve this problem, researchers adopted classical tracking-before-detect (TBD) method to aggregate information of data sequence to distinguish targets from interference [Grossi et al., 2013; Jiang et al., 2017; Zhou et al., 2019].

∗The corresponding author is Yan Chen

TBD accumulates trace-level measurement from trace proposals, and detects true target trajectory according to handcrafted shape/motion model. With deliberately designed model, TBD can accurately distinguish target trajectories from false alarms. As depicted in Figure 1, in constrained scenarios involving a few individuals moving in empty spaces, TBD achieves satisfying detection/tracking accuracy. However, in practical unconstrained scenarios involving significant human targets acting randomly in the space with various obstacles, TBD suffers from performance degradation. This is due to the fact that TBD relies on handcrafted signal model, which is difficult to handle time-varying distribution of target reflections in complex indoor scenarios. To address this problem, we build an end-to-end learning based TBD model which utilizes strong modeling capability of deep models to further unleash the power of TBD. This involves resolving two key challenges.

The primary challenge is how to achieve TBD through deep models. To aggregate temporal information, our NeuralTBD learn to accumulate inter-frame information with the help of trace center supervision, and then predicts trace offsets at each position to inference trace proposals. Specifically, NeuralTBD first extracts frame-wise deep features to adapt to the time-varying reflection distribution and motion pattern of targets. After that, it predicts per-pixel probabilities for trace centers and temporal offset sequences to generate trace proposals. Final traces are then extracted by filtering out false positives from these proposals.

The second challenge lies in the absence of a public dataset for unconstrained indoor human tracking. The main reason is that RF signal is not human readable and hard to annotate, especially in scenes with severe interference. To address this, we present the RF-UNIT (RF-based Unconstrained Indoor Tracking) dataset. It contains 4,030,880 radar heatmaps (about 56 hours) collected under 6 different office scenes with at most 19 individuals performing daily life activities. To alleviate the workload of manual labeling, we design an annotation algorithm, utilizing well-developed vision-based methods to annotate RF heatmaps. This allows us to generate full tracking annotations for the RF-UNIT dataset. We believe that the release of RF-UNIT would encourage more innovations on RF-based sensing. The main contributions of this paper can be summarized as follows:

1. To the best of our knowledge, we are the first to tackle the RF-based human tracking problem under unconstrained indoor scenarios. By leveraging the advancement of deep models and TBD, we expand the scope of RF-based tracking and provide novel insights of aggregating target information for real-world applications.
2. We propose NeuralTBD, which is a brand-new learningbased tracking-before-detect framework, utilizing deep models to adapt to time-varying target reflection distribution while preserving the temporal information aggregation capabilities of TBD. NeuralTBD realizes a 70% improvement in performance compared to conventional TBD methods.
3. We present the RF-UNIT dataset, which encompasses million-level radar heatmaps of at most 19 individuals in multiple different scenarios. To our knowledge, RFUNIT is the first fully-annotated large-scale RF dataset for indoor human tracking in unconstrained indoor scenarios.

2 RelatedWorks

2.1 Tracking-Before-Detect

In contrast to detect-then-track techniques that typically impose frame-wise detection on input data to determine the presence of target and then perform tracking. TrackBefore-Detect (TBD) methods engage with either raw data or minimally processed data and output target trajectories. TBD methods can be classified into single-frame recursive TBD (SFR-TBD) method and multi-frame TBD (MF-TBD) method.

SFR-TBD methods estimate target states at each time step by sequentially predicting and updating intermediate parameters which are then used to construct target trajectories. Typical algorithms include particle filters [Garcia-Fernandez et al., 2013], histogram-PMHT [Davey, 2014], and random finite set algorithms [Hoseinnezhad et al., 2012]. MF-TBD [Jiang et al., 2017; Zhou et al., 2019] methods accumulate target energy along physically feasible trajectories between multiple consecutive frames achieving superior tracking performance under interference. Typical MF-TBD approaches include the Hough transform [Moyer et al., 2011], maximum likelihood probabilistic data association (ML-PDA) [Ciuonzo et al., 2014], velocity matched filtering [Zhou and Wang, 2019], and dynamic programming (DP) [Zhang et al., 2021].

Despite yielding promising results, TBD methods still lack robustness in handling complex scenarios due to handcrafted shape/motion model priors, resulting in degraded performance in unconstrained indoor tracking scenarios.

2.2 Learning-Based Tracking

Existing learning-based tracking algorithms can be classified into separate detection and embedding (SDE) style and joint detection and embedding (JDE) style. SDE algorithms [Wojke et al., 2017; Wojke and Bewley, 2018] separates the tasks into detection and embedding, while JDE models [Zhang et al., 2020b] learning detection and embedding simultaneously within a shared neural network. Building upon high-accuracy detectors, such as [Ren et al., 2017; Redmon et al., 2016; Zhu et al., 2020], SDE style algorithms dominate learningbased tracking. Both SDE and JDE style tracking methods fundamentally follow the tracking by detection pipeline which highly relies on the performance of detectors.

Although remarkable results have been achieved by learning-based tracking methods, directly adopt them into our situation is infeasible. Specifically, in unconstrained human tracking scenarios using RF signals, severe interference lead to significant false alarms which pose challenges in framewise detection of targets. Therefore, we proposed to utilize the advancement of TBD and the strong modeling capability of deep models to aggregate temporal information and achieve accurate tracking in unconstrained scenarios.

2.3 RF-Based Dataset

To advance the development of learning-based approaches in wireless sensing, efforts has been made in building RF dataset of various applications. Examples include object detection [Caesar et al., 2020; Wang et al., 2021b], action recognition [Singh et al., 2019; Sengupta et al., 2020], pose/mesh prediction [Sengupta et al., 2020; Xue et al., 2021; Chen et al., 2022; Wu et al., 2022], gait [Meng et al., 2020], gesture [Palipana et al., 2021], and rehabilitation [An and Ogras, 2021; An et al., 2022]. Existing RF dataset focused on constrained scenarios involving limited number of individuals performing scheduled actions in an empty space without obstacles. However, things are different in practical unconstrained indoor sensing situation. The presence of dozens of individuals and obstacles poses great challenges in sensing tasks.

To facilitate the development of learning-based methods in unconstrained scenarios, we collect RF-UNIT, which as we know is the first large-scale RF datasets for unconstrained indoor human tracking scenarios. RF-UNIT dataset comprises 4,030,880 radar heatmaps collected under six office scenes with at most 19 individuals, each paired with corresponding tracking annotations. We hope RF-UNIT can offer valuable insights for designing RF-based deep learning models.

3 NeuralTBD

As depicted in Figure 2, NeuralTBD is an end-to-end, threestage, learning-based TBD framework that comprises a feature extraction module, a temporal accumulation module, and a trace regression module. It takes RF heatmap sequence of input and fits targets' shape/motion model with feature extraction module. And then, it utilizes temporal accumulation module to aggregate temporal information and further augment targets features. At last, it adopts trace regressionmodel to predict trace proposals followed by thresholding to eliminate false positives and produce final tracking results. In the following subsections, we provide an in-depth exploration of each components.

3.1 Feature Extraction

The biggest problem brought by interference in wireless sensing is multi-path effect, which is a fundamental and challenging problem in RF-based sensing applications. It becomes more challenging in unconstrained scenarios (involving multiple individuals acting in an obstacle-rich space), where the reflections from multi-paths can be stronger than that from targets. There are generally two kinds of multi-path interference, static multi-paths and dynamic multi-paths, with different characteristics. Static multi-paths refer to reflections from stationary objects in the environment, whereas dynamic multi-paths are caused by the movement of individuals, as depicted in Figure 3. To alleviate static multi-paths, we leverage the distinct characteristics of reflected signals from moving targets and static objects in the time domain.

Concretely, the reflections from moving individuals vary over time, while the reflections of static multi-paths remain consistent. Assuming we have a heatmap sequence S′ = {s′t }, where t is the time index, we can remove static multipaths by subtracting consecutive frames in time, as represented in the following equation:

On the other hand, signals to and from targets may be occluded, refracted and scattered by other moving objects in unconstrained scenarios, leading to significant alterations in the distribution of reflections. This presents a formidable challenge for TBD methods in accurately identifying target traces due to limited modeling capability of handcrafted shape/motion priors. To overcome this challenge, we propose to utilize the modeling capability of deep models to adapt to the dynamic changes of targets' reflections.

Specifically, given an input heatmap sequence S ∈

RH×W×T , we adopt a multi-layer convolution network with skip connections as backbone extractor E, producing framewise features ht ∈ R HR × WR ×C for each frame st, where R is the down-sampling ratio. These features are then passed through a compression network C, which transforms highdimensional but sparse features into low-dimensional and dense features to condense target information and outputs a compressed feature sequence H′ . Therefore, our feature extraction module can be represented as

3.2 Temporal Accumulation

Besides the dramatic changes in the target's reflections, severe dynamic multi-paths also result in significant false alarms with identical reflection distributions as targets, making them difficult to distinguish.

As the number of individuals increases, the occlusion and refraction of signal is increased due to mutually interference between moving individuals which introduces greater noise and intensifies the false-alarms issues. The main difference between true targets and false alarms is their continuity over time. Concretely, target positions change continuously due to limited moving speed. However, the positions of false alarms undergo rapid and erratic variations, influenced by the arrangement of objects within the environment.

TBD aggregates temporal information to distinguish target measurement from noise. Inspired by that, we propose a learning based temporal aggregation module to amplify target feature and suppress noise. Our temporal accumulation module comprises a temporal accumulator TA and a propose head P. The temporal accumulator integrates target information over an extended time period through multiple Long-Short-Term-Memory (LSTM) layers. The propose head employs multiple convolution layers to predict a confidence map, indicating potential trace centers.

Specifically, given the compressed feature sequence H′ , we first pass it through TA followed by mean operation across LSTM layers to obtain temporal-fused sequence features ˆ Y ′ . To supervise the temporal accumulation and target augmentation process of TA, we introduce propose head P , generating a confidence heatmap ˆ Y ∈ R HR × WR . Each ˆ Yx,y = 1 corresponds to a potential trace center, while ˆ Yx,y = 0 represents the background. Meanwhile, we put each scaled ground-truth trace centers onto a heatmap Y ∈ [0, 1] HR × WR using a Gaussian kernel followed by the element-wise maximum between any overlapped gaussians.

Our temporal accumulation module can be represented as

To address the significant imbalance between the number of positive and negative samples, we take the focal loss, noted as Lc, as training objective of P .

To recover the discretization error caused by the output stride R, we additionally predict an offset ˆO ∈ R HR × WR ×2 for each candidate trace center. The training objective of offset prediction can be expressed as

By supervising the confidence map ˆ Y generated by TP with ground-truth confidence maps Y , the temporal accumulation module will learn to distinguish regions containing target motions from those not by aggregating and augmenting target information in time domain.

3.3 Trace Regression

Build upon the temporally aggregated features, we are able to inference per-position trace proposals using trace regression module. Our trace regression module composed of multiple convolution layers, to predict the positional offset relative to trace center at each timestamp. Specifically, our trace regression head takes compressed features ˆ Y ′ ∈ R HR × WR ×C as input, predict positional offsets Q ∈ R HR × WR ×T×2 for traces centered at each position at each timestamp, where T denotes the length of input sequence, and the last dimension indicates horizontal and vertical offsets, respectively. We adopt L1 loss as training objective of trace regression module, denoted as Lr. After training is finished, we threshold trace proposals according to predicted confidence scores to remove false alarms and output target traces.

The overall objective is formulated as follows:

Leveraging the modeling capacities of deep model and the advancement of TBD, NeuralTBD learn to aggregate temporal information to overcome the severe interference without handcrafted efforts, and effectively predict target traces.

4 Dataset

Learning-based methods show promise to outperform traditional methods but are often limited by substantial data needs. While many RF-based datasets have been introduced , they typically cover a narrow range of scenarios with few human subjects, leaving a gap in data for unconstrained indoor environments.

Addressing this, we introduce RF-UNIT, a large-scale RF dataset for human tracking in diverse, unconstrained indoor settings. RF-UNIT comprises 4,030,880 radar frames from six office environments and includes scenes with up to 19 individuals, surpassing other public datasets that typically contains only single individual. We provide an extensive comparison with other RF datasets in Table 1. As can be seen in Figure 4, RF-UNIT is specifically tailored for real-world office environments, featuring a variety of obstacles and materials, including wood and metal, that affect signal propagation and cause notable interference. We detail individual counts per scene in Figure 6 as well as dynamic/static counts per scene in Figure 5. Most of the heatmaps in RF-UNIT contains 5-15 individuals. RF-UNIT stands as the first dataset of million-level scale for multi-person RF tracking in unconstrained scenarios.

We will introduce the hardware setup, data collection, data processing and automatic annotation methods next.

4.1 Hardware Settings

We use a TI MMWCAS-RF-EVM FMCW radar with 12 transmitters and 16 receivers offering 1.4-degree horizontal resolution, working at 77GHz and 1.23GHz bandwidth, mounted at 2.2m height. To minimize human annotation efforts, we employ an 18-node Raspberry Pi camera network, calibrated with Zhang's 2000 algorithm [Zhang, 2000], to automatically annotate data aided by vision-based techniques.

4.2 Data Collection

Deploying the well-calibrated multi-camera system in new environments can be highly time-consuming. To build a diverse dataset efficiently, we re-positioned our radar within a pre-setup scene, simulating varied RF environments and reducing the time-intensive deployment of multi-camera system. We leveraged NTP for millisecond-level synchronization between the radar and camera systems, with TCP for timestamp signal exchanges. With the cameras running at 10 fps and the radar at 20 fps, we conducted 112 data collection from six unique locations at different time of a day, each lasting 30 minutes. The total collection spans 36 days.

4.3 Data Processing

We perform signal processing algorithm to transform RF signals captured by FMCW radars into heatmap sequences. Inspired by [Zhang et al., 2018; Zhang et al., 2019b; Zhang et al., 2021], we compensate the phase shift and combine the signals of different antennas and frequencies, which coherently superimposes the signals from specific locations while suppressing the signals from other locations. In our tracking situation, we mainly concern about signals on horizontal plane. Specifically, RF signals from a specific location (x, y) can be extracted using the following equation:

where a denotes the amplitude of signal, sk,m,t denotes the k-th sample of FMCW sweep on the m-th antenna at time t, λk is the signal wavelength of the k-th sample, dm(x, y) denotes the round-trip distance from the transmitting antenna to location (x, y) and back to the receiving antenna.

4.4 Automatic Annotation

The massive collection of over 4 million RF frames, coupled with the severe interference encountered in unconstrained environments, makes manual annotation a daunting and timeconsuming task.

To deal with this problem, we developed an automatic annotation algorithm composed of three key stages: cameraspecific head detection, cross-camera triangulation, and clustering. Concretely, we employed YOLOv4 [Bochkovskiy et al., 2020], trained on the crowd-human dataset, for head detection in footage from each camera. The centers of these detections are projected as rays emitting from respective cameras. By solving intersection points of rays from different cameras, we are able to estimate 3D human positions. Lastly, we refine these estimates by applying mean-shift clustering to mitigate noise followed by a Kalman Filter to construct target trajectories. We conduct manual annotation corrections to uphold the precision of the tracking annotations.

As shown in Figure 4, our dataset is collected under unconstrained practical indoor office scenarios, encompassing challenging situations, such as diverse obstacles, varying numbers of people (up to 19), overlapping between targets, and complex human activities. We believe our RF-UNIT dataset will aid in the advancement of learning-based methods of wireless sensing.

Table 3: Ablation Experiments

5 Experiments

5.1 Implementation Details

In this section, we present the evaluation results of NeuralTBD on RF-UNIT dataset. We first preform group-wise shuffle on RF-UNIT and divide data into train, validation, and test subset, following 8:1:1 ratio. All evaluations are reported on test sets. NeuralTBD is designed to deal with RF heatmap sequence within a fixed time-window. Therefore, we adopt sliding window strategy to deal with long sequence. At each step of this process, we retain trace results that exhibit a higher confidence score and/or a lower Intersection over Union (IoU) with pre-existing traces, ensuring both accuracy and minimal redundancy. We employ the Adam optimizer with initial learning rate of 1.0×10-2 and weight decay of 0.05. During the training process, we adopt a step-based learning rate decay strategy. All experiments are conducted on a single NVIDIA A100 GPU with a batch size of 16.

5.2 General Performance

To showcase the performance of our NeuralTBD on the RF-UNIT dataset, we conducted comparative experiments against baseline methods, including both TBD and learning based methods:

• MF-TBD [Grossi et al., 2013]: MF-TBD associates CFAR segmentation results across frames according to IoU and shape similarity to propose traces. And then it retains those with trace energy higher than a threshold.
• MKCF-TBD [Zhou et al., 2019]: MKCF-TBD improves MF-TBD using kernel methods which better models target distribution of reflections.
• MTrack [Zhang et al., 2020a]: MTrack is a graph-based TBD method. It constructs a directed graph based on CFAR segmentation results and subsequently solves for the shortest path to accomplish tracking tasks.

• RODNet [Wang et al., 2021a]: RODNet is a learningbased radar object detection method. We performs kalman filter on the detection results of RODNet to output trace predictions.

Table 2 shows that our NeuralTBD significantly outperforms traditional methods, achieving a higher AP of 0.885, compared to the substantially lower scores of MF-TBD (0.521), MKCF-TBD (0.333), MTrack (0.378), and RODNet (0.842). These results underline the limitations of conventional TBD techniques in detecting human targets indoors.

Furthermore, our method achieves a substantially higher IDF1 score of 44.8%, exceeding other methods. This signifies that NeuralTBD undergoes fewer trace mismatches and ID switches when processing long sequences. In addition, the MOTA value of 0.420 illustrates that NeuralTBD provides superior target-locating capabilities. These metrics highlight the efficacy and advancement of our learning-based TBD approach.

The qualitative results in Figure 7 further demonstrate the advantages of our method over baseline approaches. Baselines exhibit substantially more false alarms, with long target traces fragmented into shorter segments. In contrast, our approach produces fewer false alarms and successfully tracks targets over extended periods. This highlights the effectiveness of our method in leveraging deep models' robust representational capabilities to differentiate targets from noise and clutter. Moreover, it fully capitalizes on the benefits of TBD to accomplish long-term tracking in unconstrained settings.

5.3 Ablation Study

In this subsection, we provide ablation results on temporal accumulation module (denoted as TA) and on different input sequence length. Refer to Table 3 for the detailed experimental outcomes.

Temporal Accumulation Module

To assess the effectiveness of temporal accumulation module, we substituted it with CFAR segmentation followed by kalman filter to inference trace proposals, while keeping the other modules unchanged.

The experimental results demonstrate a substantial decrease in performance when the temporal accumulation module is omitted for all input sequence lengths. In particular, the AP score drops from 0.885 to 0.455, and the MOTA value decreases from 0.420 to 0.102 with an input sequence of 24 frames. This underscores the significant enhancement in trace localization achieved through temporal accumulation. Overall, these outcomes validate the effectiveness of our NeuralTBD approach.

Time Domain Analysis

Constrained by hardware limitations, our NeuralTBD accumulates information within a fixed time window, termed the accumulation window, to predict target motions. To further analyze the impact of the accumulation window length, we conduct experiments varying the input sequence duration. As shown in Table 3, NeuralTBD exhibits significant performance gains as the input sequence lengthens. For example, when the input sequence increases from 6 to 24 frames, the AP, MOTA, and IDF1 values rise from 0.852, 0.254, 0.246 to 0.885, 0.420, 0.448, respectively. This implies longer accumulation windows yield enhanced tracking accuracy.

However, NeuralTBD experiences a performance bottleneck once the sequence surpasses 24 frames. We assume this relates to the long-term modeling capacity of NeuralTBD.

6 Conclusion

This study explores human tracking of unconstrained indoor environments, introducing NeuralTBD which builds a brandnew learning-based TBD framework, utilizing strong modeling capability and advancement of TBD to aggregate temporal information and address multi-path effects. To train NeuralTBD, we also collected RF-UNIT, the first large-scale public RF dataset for unconstrained indoor human tracking, comprising 4 million heatmaps with at most 19 individuals collected from six different scenarios. NeuralTBD realizes a 70% improvement in performance compared to conventional TBD methods. RF-UNIT as well as NeuralTBD are valuable resources for propelling RF-based tracking and downstream applications, empowering the community to develop robust solutions for practical wireless settings. The dataset and code will be public.

Acknowledgements

This work was supported by National Key R&D Programmes under Grant 2022YFC2503405, National Natural Science Foundation of China under Grant 62201542, 62172381 and 62302471, the Fundamental Research Funds for the Central Universities, the fellowship of China Postdoctoral Science Foundation under Grant 2023M743401, and the Postdoctoral Fellowship Program of CPSF under Grant GZC20232565.