Bridging LiDAR Gaps: A Multi-LiDARs Domain Adaptation Dataset for 3D Semantic Segmentation

1 Introduction

3D semantic segmentation is a foundational task in robotics [Grigorescu et al., 2020; Xia et al., 2021a] and autonomous driving [Shen et al., 2023; Li et al., 2023]. Compared with 2D images, point clouds collected by LiDAR offer accurate 3D geometric properties and depth insights [Xia et al., 2024]. Over the past years, a series of learning-based 3D segmentation methods [Hu et al., 2020; Thomas et al., 2019; Huang et al., 2024] achieve excellent performance when training with extensively annotated datasets (e.g., SemanticKITTI [Behley et al., 2019], nuScenes [Caesar et al., 2020], and SemanticPOSS [Pan et al., 2020]). However, these SOTA segmentation methods often rely on a crucial assumption: the training and testing datasets should have similar data distributions.

* Corresponding author

In practice, the quality and characteristics of scanned point clouds differ significantly, influenced by various LiDAR sensor configurations and physical environments. Notably, the differences in the disparities in laser line count, field of view, and detection range especially pose substantial challenges for cross-domain research in 3D point cloud domain adaptation.

Current 3D domain adaptation methods [Yi et al., 2021; Langer et al., 2020] leverage existing datasets [Sun et al., 2020; Pan et al., 2020] as benchmarks to validate their effec-

are typically collected by different authors using diverse sensors in various scenes and annotated according to distinct rules. For example, SemanticKITTI [Behley et al., 2019] provides point cloud data with 64-beam and annotations for 19 semantic categories, while nuScenes [Caesar et al., 2020] is collected using a 32-beam LiDAR and annotated with 16 categories. In addition, nuScenes cover multiple road scenes in different countries to provide rich samples and add diversity for 3D semantic segmentation tasks but not semantic segmentation domain adaptation. Consequently, such benchmarks amalgamate multiple factors influencing domain gaps, posing challenges for meticulous individual analysis. Recently, some studies [Wu et al., 2018; Xiao et al., 2022b] utilize virtual engines or simulation methods to rapidly acquire annotated point cloud data, concurrently presenting adaptive approaches from virtual to real data. While such methods effectively alleviate the burden of acquiring labeled data, the domain gap between synthetic and real data remains highly challenging. From the scenario above, there is still a lack of datasets specifically designed to address domain adaptation issues concerning cross-sensor and cross-scenario.

To address this issue, we introduce a Multi-LiDAR Domain Adaptation Dataset for 3D semantic Segmentation (MLDAS). The dataset is equipped with 3 typical types of multi-beam LiDAR: 32-beam, 64-beam, and 128-beam. LiDARs are time-synchronized and calibrated to capture the same scenario, mitigate the impact of dynamic objects on LiDAR scans. Utilizing a vehicle-mounted platform, we collected 10,625 frames of data per LiDAR across two distinct scenes, with 6,395 frames on campus and 4,230 frames on the street. We performed detailed point-wise annotations for 14 categories on the 128-beam data and subsequently transferred the labels to the 64-beam and 32-beam data. Thanks to mIOU

precise sensor calibration and time synchronization, the label transfer process significantly reduced the human effort and time required for data annotation. Based on this foundation, we will elaborate on the dataset's capacity to investigate the impact of cross-sensor and cross-scene factors on semantic segmentation domain adaptation.

Building upon MLDAS, we propose a method based on Hierarchical Segmentation network with Spatial Consistency (HSSC) to investigate the cross-sensor and cross-scenario challenge in 3D semantic segmentation domain adaptation (DA). Unlike existing methods [Xie et al., 2020; Xiao et al., 2023a] that achieve consistency constraints between different views through data augmentation, our approach utilizes sensor calibration relationships and synchronization to establish corresponding relationships in the same spatial-temporal context across extra LiDAR data. It allows us to focus on the search between different sensors, mitigating the impact of environmental variables. Employing such consistency constraints, we apply the typical mean-teacher architecture to transfer knowledge from the source to the target sensor. In unsupervised domain adaptation (UDA) tasks, experimental results demonstrate the effectiveness of our HSSC in crosssensor knowledge transfer compared to previous SOTA methods. Furthermore, to assess the generalization capability of our HSSC, we collect extra data from a location distinct from the Campus and Street. This supplementary dataset serves as a transition domain for cross-domain training of the model. In the absence of target domain data, we verify the effectiveness of our HSSC in UDA.

To summary, our main contributions include:

• We introduce a Multi-LiDAR Domain Adaptation Dataset for 3D semantic Segmentation(MLDAS), including 3 types of multi-beam spinning LiDARs scanning 2 scenes with point-wise annotations. MLDAS will establish a robust groundwork for exploring uncharted territories in cross-LiDAR and cross-scenario domain adaptation in LiDAR point cloud segmentation research.
• We investigate the significant factors underlying the domain gap in cross-sensor and cross-scenario, leading to the design of HSSC, a framework capable of transferring knowledge from the source sensor to the target sen-
• sor, effectively mitigating the domain gap.
• We benchmark various UDA algorithms for LiDAR semantic segmentation, providing comprehensive baselines to support cross-sensor and scenario segmentation research.

2 RelatedWorks

2.1 LiDAR Segmentation Datasets

The recent emergence of outstanding learning-based LiDAR semantic segmentation methods [Zhu et al., 2021; Tang et al., 2022; Unal et al., 2022] has sparked a demand for extensive semantic annotation data, leading to the introduction of numerous datasets. Currently, widely utilized datasets include SemanticKITTI, nuScenes, Waymo [Sun et al., 2020] and semanticPOSS [Pan et al., 2020]. These datasets offer rich varieties of LiDAR types and diverse scene data, significantly advancing the field of semantic segmentation tasks. However, employing these datasets directly for domain adaptation introduces complications related to various influencing factors such as cross-sensor and cross-scenario effects, hindering effective evaluation.

To address the needs of certain semantic segmentation domain adaptation tasks, specific datasets have been proposed. SynLiDAR [Xiao et al., 2022b] uses a simulation platform to simulate a 64-beam LiDAR.While it alleviates the issue of annotating real data, the gap between virtual and real data remains challenging to overcome. Since existing datasets are collected under normal weather conditions, they are not conducive to studying semantic segmentation under adverseweather conditions. SemanticSTF [Xiao et al., 2023b] conducts an adverse-weather dataset to facilitate research on semantic segmentation adaptation in all-weather conditions. Despite the availability of numerous datasets for 3D semantic segmentation, there is still a lack of datasets in real-world scenarios for semantic segmentation domain adaptation while decoupling the influence of scenes and LiDAR types.

2.2 Domain Adaptation in 3D Semantic Segmentation

Earlier research [Zhao et al., 2021; Wu et al., 2019; Jiang and Saripalli, 2021] applied the 2D DA method to range images projected from 3D point clouds. As the development of point cloud processing networks [Qi et al., 2017; Zhou and Tuzel, 2018] progresses, researchers are increasingly focusing on the characteristics of 3D point clouds. Some researchers employ sparse point cloud completion method [Xia et al., 2021b] to ease sensor density variation. Data augmentation [Yun et al., 2019; Wu et al., 2020] is commonly employed in 2D images to mitigate domain gaps between diverse datasets. Inspired by this, Cosmix [Saltori et al., 2022] and Polarmix [Xiao et al., 2022a] encourage the model to learn cross-domain representations by blending source and target data differently. Similarly, PCT [Xiao et al., 2022b] adapts synthetic point clouds to match the appearances and sparsity of real point clouds. SCT [Xiao et al., 2023a] achieves domain adaptive segmentation by enforcing spatial consistency constraints on the target domain data before and after data augmentation. Distinguishing from it, our approach avoids the necessity for intricate augmentation techniques. Instead, we directly leverage the spatialtemporal consistency constraints among real LiDAR datas to learn cross-sensor&scenario representations.

3 MLDAS Dataset

3.1 Sensors Setup

MLDAS comprises point cloud data from three mechanical spinning LiDARs with various beams, including: (1) Hesai XT32, a 32-beam low resolution LiDAR (denoted as XT32). (2) Ouster OS1-64, a 64-beam LiDAR (denoted as OS64). (3) Ouster OS1-128, a 128-beam high resolution LiDAR (denoted as OS128). These LiDARs cover the common application and research. In addition, we install a GNSS/IMU device (M39) to obtain the ego-motion pose. Detailed specifications for these sensors are described in Table. 1.

Figure 2 shows a rigid support system for securing the LiDARs to ensure that the sensor suites operate reliably. Different LiDAR brands have varying capabilities and accuracy (shown in the Supplementary Material), and each uses its own coordinate system, as illustrated in Figure 2 (b). Theses disparities facilitate our research on cross-sensor domain adaptation. The LiDAR units are mounted vertically to minimize the interference or occlusion between them during operation and to maximize coverage of a 360◦ scene. Additionally, the positions of the LiDAR centers are not aligned on a single axis to diversify collection positions.

3.2 Sensors Synchronization and Calibration

Precision Time Protocol (PTP) [Eidson and Kang, 2002] is used to synchronize time amongst three LiDAR units, reliant on GNSS clocks provided by M39 for time reference.

The synchronization mechanism ensures that the cross-sensor data is aligned properly. In addition, we also achieve precise extrinsic calibration among sensors. Specifically, MLDAS sensor calibration includes both LiDAR-to-LiDAR and LiDAR-to-GNSS/IMU calibrations. We strategically position rigid objects with regular shapes within the detection range of the LiDARs for precise calibration of LiDAR-to-LiDAR. We subsequently identify corresponding corner points and apply the Generalized-ICP [Segal et al., 2009] algorithm to derive the final external parameters. The calibration method of LiDAR-to-GNSS/IMU refers to KITTI [Geiger et al., 2012].

3.3 Scene Selection

To decouple the effects of cross-sensor and cross-scenario in domain adaption, we collect data in two representative scenarios: campus and urban street. These scenarios are vastly different in point number and distribution of some categories. For example, on campus, there will be fewer cars on campus, more bicycles, and more vegetation, as well as a relatively dispersed pattern of buildings. The distance between dynamic categories and the collection vehicle is random on campus due to the lack of strict traffic rules. However, in city streets, there is a relatively large flow of traffic and people, and there are densely populated areas on both sides of the street. Traffic rules ensure that people and vehicles are able to move within the prescribed range. Consequently, there is a significant shift in the domain between the two scenes. Compared with existing multi-scene collected dataset [Xiao et al., 2021; Caesar et al., 2020], MLDAS covers a wider area, which guarantees a comprehensive and representative distribution of data. This methodology is intended to provide ample and stable domain gaps for subsequent domain adaptation research.

3.4 Annotation

MLDAS provides point-wise semantic segmentation annotations for three types of LiDAR data. Our methodology commences with the annotation of the densely populated OS128 data to achieve efficient and high-quality annotations. Subsequently, we capitalize on sensor calibration and synchronization relationships to propagate these annotations to the other two LiDAR datasets. We now elaborate on the annotations generation and propagation of MLDAS.

In the annotation generation stage, inspired by SemanticKITTI [Behley et al., 2019], we build the point cloud map from OS128 data for efficient labeling. Specifically, we initialize the map using the poses provided by GNSS/IMU. Then, we employ an offline SLAM algorithm [Zheng et al., 2022] to enhance map quality. Finally, we annotate 14 categories for semantic segmentation evaluation.

In the annotation propagation stage, we employ a temporalspatial constrained method to transfer the annotations from OS128 to the other two LiDARs data. The first step is to transform the XT32 data into OS128 coordinate system using calibration relationships and position information obtained through SLAM. Our subsequent step is to search for corresponding points in the OS128 map for the XT32 points using the nearest neighbor search algorithm [Bentley, 1975]. This process establishes a spatial correspondence between the two LiDAR data. Due to the presence of dynamic objects in the scene, points within the candidate set may represent different categories at different times but occupy the same location. We eliminate points from the candidate set that do not align with the temporal information of XT32 data by leveraging the temporal synchronization relationships between LiDARs. We assign the label of the nearest neighbor OS128 points for the remaining candidate points to XT32 points. OS64 data annotation follows a similar procedure, which is not repeated here. We ensure the accuracy and consistency of semantic annotations across different LiDAR data sets by calibrating and synchronizing devices, providing a foundation for further research on domain adaptation.

3.5 Dataset Statistics

MLDAS collects 31,875 frames of point cloud data and annotates 14 categories in total. We partition each type of LiDAR data from the two scenes into training, validation, and test sets in a ratio of 8:1:1.

Distribution of Point Numbers. The differences in the quantity and distribution of point clouds from different LiDARs are evident, as illustrated in Figure 3 (a). The data volume of OS128 is significantly higher than that of other LiDARs, posing a challenge for cross-sensor domain adaptation. To better understand the distribution of different LiDAR data, we visualize them in Figure 1 (a). We can observe that the density of three types of LiDAR points decreases with the increasing distance from the sensor. However, due to differences in the field of view and the number of lines, the distribution characteristics of the data are markedly different.

Distribution of Categories. Figure 3 (b) illustrates the distribution of points for 14 semantic categories in MLDAS, showing significant variations in the point count among different categories. Clearly, Vegetation and building have much higher point counts than others. In addition to the diversity in quantity, the spatial distribution diversity of categories is crucial for DA research. Therefore, we analyze the distribution of labels around the sensors for car, pedestrian categories in both scenes through a polar log-scaled density map, as shown in Figure 4. As mentioned in Section 3.3, traffic rules constrain the distribution of pedestrians and cars in the streets, resulting in higher density along the direction of vehicle movement. In contrast, pedestrians on the campus exhibit a more uniform distribution.

4 HSSCMethods

4.1 Definition

According to UDA setup, we have point cloud from a source domain dataset Dsrc = {xisrc , yisrc}Nsrc i=1 which contains Nsrc = |Dsrc | point clouds xsrc with point-wise label ysrc and an unlabeled target domain Dtgt = {xitgt}Ntgt i=1 . Because MLDAS incorporates multiple LiDARs, for clarity and distinction, we refer to the data, excluding the source and target domains, as the intermediate domain Dint = {xiint}Nint i=1 . The intermediate domain aims to establish connections between the source and target domains. In this study, Dint and Dsrc share the same LiDAR but distinct scenarios. In contrast, Dint and Dtgt utilize different LiDARs while scanning the data simultaneously. As a result, Nint = Ntgt and the timestamps of xiint and xitgt are the same.

4.2 Points-wise Matching and Spatial Consistency

Based on MLDAS, we propose a concise and effective method for point-wise correspond matching between two frames of point clouds collected by different sensors. Initially, leveraging the time synchronization within MLDAS, we identify the point clouds xtint and xttgt from Dint and Dtgt at the time t. Subsequently, employing the calibration relationship T , we transform the point cloud xttgt to the coordinate system of xtint . Finally, We utilize the nearest neighbor algorithm [Bentley, 1975] to obtain the nearest correspond point of xttgt in xtint with a distance below a predefined threshold τ. We record the indexes of the matching pairs in the array as a correspondence mapping M t . A pair of correspond points reflect the modeling of the same object in spatial by different sensors, possessing consistent contextual information. Building on this spatial consistency, we train the model to adapt to diverse sensor data.

4.3 Hierarchical Segmentation

Utilizing the intermediate domain, we propose the Hierarchical Segmentation Network with Spatial Consistency (HSSC) learning strategy. Our learning paradigm build up on the teacher-student framework with two same backbone of 3D point cloud segmentation, as illustrated in Figure 5. The process of our HSSC can be divided into two hierarchical learning level:

1. Adapting from source Dsrc to intermediate domain Dint. We feed xint into the teacher model. Based on knowledge of Dsrc, the network output the logits predictions pint, which applied as the supervised signals in the next level.
2. Adapting from intermediate Dint to target domain Dtgt. We sequentially input xtgt into the student model to generate logits predictions ptgt. The adaptation can be achieved by a spatial-consistency loss that enforce ptgt to be consistent with pinit.

4.4 Network Architecture and Training

Both networks share the same 3D semantic segmentation backbone [Choy et al., 2019]. For the source domain, we apply cross entropy loss Lseg to optimize the student network and train model with labeled data, F(·) refers to the network:

To narrow the gap between the target and source domains, we construct an intermediate domain based on MLDAS, as described in Section 4.1. We employ a self-supervised training approach for the intermediate and target domain data. Initially, we select a pair of point cloud data {xtint , xttgt} at time t. We feed xtint data to the teacher network, obtaining prediction logits ptint = Ftea(xtint ). Similarly, we obtain the logits prediction pttgt = Fstu(xttgt ) through the student network. Subsequently, based on the mapping relationship obtained M t in Section 2.2, we derive the correspond prediction logits ˆ ptint and ˆ pttgt . This operation is denoted as Ψ(·) :

where |ˆ ptint | = |ˆ pttgt |. We then calculate the cosine similarity loss between ˆ ptint and ˆ pttgt :

where ˆ pt,i int and ˆ pt,i tgt is the prediction logits of the i-th correspond points between xtint and xttgt . The total objective loss L defined as:

In the end, We deploy the Expoential Moving Average (EMA) algorithm to update the weights of the teacher model:

where α is a smoothing coefficient hyperparameter.

5 Experiments

5.1 Experiments Setups andMetrics

Setup. We design three experiment setups for 3D semantic segmentation UDA: (1) cross-sensor&scenario, (2) crosssensor, (3) cross-scenario. All experiments are implemented in Pytorch [Paszke et al., 2019] and MinkowskiEngine [Choy et al., 2019] on a single NVIDIA RTX 3090 GPU. For a fair comparison, all methods for experiments select the MinkowskiNet [Choy et al., 2019] segmentation architecture as the backbone. The voxel size of the network is set to

0.05m. We train the model on the source domain data using the SGD optimizer and cross-entropy loss function for 30 epochs. Subsequently, we select the best pre-trained model based on evaluating the source validation dataset and re-use the same model for all methods. We select five baseline methods, including source-only, ST [Zou et al., 2018], CosMix [Saltori et al., 2022], PolarMix [Xiao et al., 2022a], and SCT [Xiao et al., 2023a]. Source-only means applying the pre-trained model to the target data without any extra finetuning. ST is a domain adaptation method for 2D semantic segmentation, and others are used in 3D point clouds. As SCT has yet to release its source code, we replicated the method based on the information provided in the paper. We utilize target and intermediate domain data for the baselines above to ensure a fair comparison.

Evaluation Metrics. Following previous research [Behley et al., 2019], we employ per-class Intersection-overUnion (IoU) and mean IoU (mIoU) metrics to evaluate the model on 14 categories. In the following sections, we will describe the experimental settings under different conditions.

5.2 Experiments on Cross-Sensor&Scenario

The simultaneous influence of two factors on DA is the most common scenario. Therefore, this experimental setup most directly reflects the effectiveness of UDA methods. We conduct two UDA experiments, considering two LiDAR types (XT32 and OS128) and two scenes (Campus and Street): (1) Campus(XT32)→Street(OS128), (2) Campus(OS128)→Street(XT32). We adapt the model from the source to the target domain. In addition, we apply Street(XT32), Street(O128) as intermediate domain individually.

Comparisons with SOTA. Table 2, 3 present the detailed experimental results. Our method consistently outperforms all state-of-the-art UDA methods on mIoU across different experimental settings, significantly achieving an improvement of +14.67 mIoU over the suboptimal result. These findings demonstrate the ability of our approach to robustly learn knowledge from different sensors and scenes. Our approach maintains commendable performance when migrating from a high-beam LiDAR to a low-beam LiDAR.

Table 6: Adaptation results on Street(OS128)→Campus(XT32). Performances of our approach trained on different datasets. "intermediate" refers to Campus(OS128). "Place C" shares the same LiDAR type with the source and target domain, but the data is scanned from other places.

5.3 Experiments on Cross-Sensor

In this experiment, we select the Campus to conduct sensor adaptation from OS128 to XT32 data, denoted as Campus(OS128)→Campus(XT32), without introducing an intermediate domain.

Comparisons with SOTA. Analyzing the experimental results from Table 4, we can draw several conclusions: (1) The pre-trained model achieves satisfactory metrics in the target domain. (2) However, the large gap between sensors makes the improvement of existing methods insignificant and even performance degradation. (3) Our proposed method, designed to adapt to sensor variations, successfully learns the target data's characteristics and outperforms other methods.

5.4 Experiments on Cross-Scenario

In this experiment, we select XT32 data and design the experiments: Campus(XT32)→Street(XT32), and apply Street(OS128) as intermediate domain. As we exclude sensor factors, introducing additional intermediate domain data may adversely affect other methods. Hence, we conduct experiments with and without intermediate domain data for other methods and select the best results.

Comparisons with SOTA. From the results of Table 5, when adapting from Campus to Street, ST achieves the best results for its class-balance strategy. Due to the introduction of intermediate domain data (considering the massive data volume of OS128), our method inevitably incurs negative impacts. Our method still achieves results comparable to it.

5.5 Discussion

The above experiments assume that the intermediate and target domains share similar scenes and scan simultaneously. This assumption may not hold since we only have one data type for the target domain (LiDAR type is known). Sometimes, we may have no access to the source domain. Here, we will briefly examine the generalization of our approach to the above cases. First, we construct an intermediate data domain using the MLDAS multi-LiDAR system to collect data, where the LiDAR type corresponds to the source domain and the target domain. Due to the lack of intersection between Campus and Street and semantic annotation, we name the scene Place C. Under the setting of Street(OS128)→Campus(XT32), different datasets were used for training and verification on the target domain, and the experimental results are presented in Table 6. The results indicate that our method can be generalized without relying on assumptions derived from previous experiments. More ablation studies are presented in the Supplementary Material.

6 Conclusion

We introduced a new dataset, MLDAS, for 3D segmentation domain adaptation, incorporating three precisely synchronized and calibrated LiDARs and two diverse scenes. Moreover, we proposed a novel hierarchical segmentation network with spatial consistency (HSSC) to address cross-sensor and cross-scenario problems. Extensive experiments demonstrate that the proposed HSSC significantly improves the 3D segmentation performance over the state-of-the-art methods. We hope our work will inspire more investigation into the 3D segmentation domain adaptation problem.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (No.20720230033), and PDL (2022-PDL-12).

Contribution Statement

Shaoyang Chen and Bochun Yang have made equal contributions to this work.