REHEARSE-3D: A Multi-Modal Emulated Rain Dataset for 3D Point Cloud De-Raining

Abstract

Sensor degradation poses a significant challenge in autonomous driving. During heavy rainfall, interference from raindrops can adversely affect the quality of LiDAR point clouds, resulting in, for instance, inaccurate point measurements. This, in turn, can potentially lead to safety concerns if autonomous driving systems are not weather-aware, i.e., if they are unable to discern such changes. In this study, we release a new, large-scale, multi-modal emulated rain dataset, REHEARSE-3D, to promote research advancements in 3D point cloud de-raining. Distinct from the most relevant competitors, our dataset is unique in several respects. First, it is the largest point-wise annotated dataset (9.2 billion annotated points), and second, it is the only one with high-resolution LiDAR data (LiDAR-256) enriched with 4D RADAR point clouds logged in both daytime and nighttime conditions in a controlled weather environment. Furthermore, REHEARSE-3D involves rain-characteristic information, which is of significant value not only for sensor noise modeling but also for analyzing the impact of weather at the point level. Leveraging REHEARSE-3D, we benchmark raindrop detection and removal in fused LiDAR and 4D RADAR point clouds. Our comprehensive study further evaluates the performance of various statistical and deep learning models, where SalsaNext and 3D-OutDet achieve above $94\%$ IoU for raindrop detection.

1. Introduction

Autonomous driving technology has undergone significant advancements in recent years. However, the reliability and performance of autonomous vehicles in inclement weather conditions remain a significant challenge. Among these challenges, rainfall poses a particularly complex problem, as raindrops can adversely impact sensor performance, thereby hindering downstream perception tasks [1].

LiDAR (Light Detection and Ranging) sensors have become a staple in autonomous vehicles due to their high accuracy, reliable depth calculation, and long-range measurements. However, the integrity of these estimations can significantly deteriorate due to precipitation, such as raindrops, which can scatter and refract the laser beams. Precipitation characteristics, such as intensity and frequency, significantly impact the accuracy of the resulting point clouds. For instance, heavy rainfall can substantially degrade the quality of the 3D LiDAR point cloud, causing inaccurate and sparser point clouds. This, in turn, can lead to miss-detection and false positive predictions in downstream perception tasks [2,3].

To address these weather-related challenges, numerous studies have been conducted not only by introducing diverse datasets to capture the impact of rainfall on sensor readings [4,5,6] but also by identifying and filtering out precipitation-related noisy measurements (also known as de-raining), particularly those caused by raindrops in 3D point clouds [3,5,7]. However, existing annotated datasets are unimodal (using LiDAR only) with limited data density (max 64-beam scans) and lack diversity in rain intensity (see Table 1). These limitations impede the efficacy of benchmarks for point-cloud de-raining.

Table 1.

Comparison of relevant outdoor datasets with REHEARSE-3D. #points is in millions.

	#Points	#Classes	Modality	Annotation	Sequential	Weather	Rain Characteristics	Day/Night	Environment
SemanticKITTI [8]	4549	28	LiDAR-64	Point-wise	✓	Clean	-	-	Real
SnowyKITTI [9]	3940	2	LiDAR-64	Point-wise	✓	Snow	-	-	Simulated
WADS [7]	387	22	LiDAR-64	Point-wise	✓	Snow	-	-	Real
SemanticSpray [4]	526	3	LiDAR-32	Point-wise	✓	Rain	-	-	Real
WeatherNet [5]	1700	3	LiDAR-32	Point-wise	✓	Rain/Fog	✓	-	Real
REHEARSE-3D (Ours)	9200	8	LiDAR-256 ★ and 4D RADAR	Point-wise	✓	Rain/Clean	✓	✓	Real

^★ The LiDAR is a MEMS LiDAR with 256 lines.

To push the boundaries of research in this area, we present REHEARSE-3D, a novel, large-scale, and multi-modal emulated rain dataset, including high-resolution LiDAR-256 scans complemented with weather-resilient 4D RADAR (Radio Detection and Ranging) point clouds logged in both daytime and nighttime rainy conditions. REHEARSE-3D is enriched with semantic information for each LiDAR and RADAR point (see Figure 1) to facilitate research in 3D point cloud de-raining for autonomous driving applications. Furthermore, REHEARSE-3D provides precipitation characteristic information, including rain intensity, droplet size distribution, wind velocity and direction, and visibility. This information is of significant value not only for sensor noise modeling but also for analyzing the impact of weather at the point level. To the best of our knowledge, REHEARSE-3D is the first comprehensive, multi-modal, point-wise annotated, high-density point cloud dataset capturing various rain effects across day and night conditions in a controlled weather environment.

Figure 1.

A sample scene from REHEARSE-3D. Fully annotated high-resolution LiDAR and fused 4D RADAR point clouds are shown on the right. Each color represents a unique semantic class: rain, car, sprinkler, pedestrian, biker, road, targets, and background. For visualization purposes only, the corresponding RGB and thermal camera images are shown on the left.

REHEARSE-3D is the semantically enhanced iteration of the REHEARSE (adveRse wEatHEr datAset for sensoRy noiSe modEls) dataset [10] conducted in a controlled outdoor rain facility with target objects (e.g., pedestrians, cars, bikers, etc.) at fixed positions. For rain emission, REHEARSE employs a set of mobile water sprinklers to generate synthetic rain in three precipitation levels: light ($10\mathrm{mm}/\mathrm{h}$), medium ($25\mathrm{mm}/\mathrm{h}$), and heavy ($50\mathrm{mm}/\mathrm{h}$). The dataset involves an automotive-grade sensor suite, including not only high-resolution LiDARs and a 4D RADAR but also RGB and thermal cameras. We note that REHEARSE-3D provides point-wise semantic annotations only for MEMS LiDAR and 4D RADAR readings.

Leveraging REHEARSE-3D, we establish a comprehensive benchmark for de-raining fused LiDAR and 4D RADAR point clouds. We evaluate the performance of several state-of-the-art statistical and deep learning baseline models to provide an in-depth analysis. The objective is to establish strong baseline performance standards that would facilitate and advance future research efforts in 3D point cloud de-raining.

In short, our contributions are listed below:

• We introduce a new large-scale multi-modal emulated rain dataset, named REHEARSE-3D, with 9.2 billion point annotations, logged in various rain intensities in daytime and nighttime conditions in a controlled weather environment.

• Leveraging REHEARSE-3D, we benchmark various state-of-the-art denoising algorithms to de-rain the early-fused LiDAR and RADAR point clouds.

• We use the point cloud from clean weather and statistically generate synthetic raindrops to study the emulated-to-simulated domain gap.

2. Related Work

A substantial corpus of research has been devoted to multi-modal datasets for autonomous driving [11,12,13,14,15], as comprehensively outlined in [16,17]. When it comes to adverse weather conditions, datasets are scarce since some concentrate on individual modalities, such as LiDAR [7,18] or RADAR [19], or address specific tasks, such as object detection and localization, as in the case of [20,21,22], or exclude annotations for outliers, such as raindrops or snowflakes, as in [18,23], despite providing ground-truth point-wise semantic annotations for the full 3D point cloud data. Additionally, there exist synthetically corrupted datasets to simulate severe weather conditions [24,25,26], which inevitably suffer from the simulation-to-reality gap.

Regarding emulated weather datasets, TWICE [27] introduced a synthetic rain dataset using a camera and a 3D RADAR. The logged 2D rain data, however, is neither properly validated nor annotated. The work in [6] solely addresses the degradation of LiDAR and camera readings in rainy conditions, conceptualizing weather monitoring as a regression task, for instance, by estimating rainfall intensity. However, no semantic information is provided for the logged sparse 32-beam LiDAR data.

Only a few relevant studies provide detailed point-wise ground-truth annotations, even for outliers, such as raindrops and/or snowflakes, in harsh weather data [4,7,9]. WADS [7] is a real 3D LiDAR point cloud dataset with point-wise labels for falling snow and accumulated snow. This dataset, however, lacks data from other sensor modalities and recordings for rainy conditions. Likewise, SnowyKITTI [9] is a simulated snowy iteration of the real SemanticKITTI [8] dataset, coming with LiDAR-only sensor readings. SemanticSpray [4] is a real 3D point cloud dataset scanned with LiDAR-32. This dataset is based on RoadSpray [28]; however, it only contains scenes following a rainfall in which the road exhibits three distinct levels of water accumulation. The precise quantitative data concerning the rainfall is not included in the dataset. The spray effect is generated by the ego vehicle and the surrounding individual vehicles. WeatherNet [5] is the closest work to ours. It provides emulated rain and fog point cloud data logged in an indoor rain/fog chamber. This dataset exhibits a notable deficiency in sensor resolution (32-beam LiDAR), with ground-truth annotations encompassing only a forward-facing view of about 60° in a horizontal field, thus substantially diminishing the number of labeled points.

Most of these aforementioned datasets contribute to the field by providing 3D point-wise semantic annotations for sequential LiDAR scans. As shown in [2,3], such annotations are crucial in detecting and removing noisy points to further enhance downstream perception tasks (e.g., semantic segmentation and object detection) under adverse weather conditions. However, all these datasets focus only on LiDAR data with limited resolutions (max 64-beam, as in the case of WADS [7]). LiDAR point clouds are well known for being sparse, making it challenging to develop perception solutions for them. On top of that, inclement weather makes it even worse because the interaction between the laser beam and the raindrop/snow particle makes the point cloud even sparser. Such a drop in sparsity can be seen in the work using SemanticSpray [4] and WeatherNet [5]. This inevitable sparsity issue warrants the collection of a high-density point cloud.

Table 1 compares our dataset with these highly relevant counterparts. Our proposed REHEARSE-3D differs from all these existing competitors in that it is the largest annotated dataset and the only one with high-resolution LiDAR data (LiDAR-256) enriched with a 4D RADAR point cloud. We note that REHEARSE-3D also includes RGB, thermal camera, and weather sensor readings, which, however, are not semantically labeled. Like WeatherNet [5], REHEARSE-3D additionally provides rain-characteristic information at three levels ($10\mathrm{mm}/\mathrm{h}$, $25\mathrm{mm}/\mathrm{h}$, and $50\mathrm{mm}/\mathrm{h}$) for emulated rain scenes generated by sprinklers, which paves the way for grounding the computational modeling of emulated raindrops. This information is of significant value when comparing simulated and real counterparts, thereby facilitating the process of bridging the sim-to-real gap. Consequently, Table 1 conveys that REHEARSE-3D makes a significant contribution to the field by providing the largest point-wise annotated high-density point clouds that are logged in various rain intensities in both daytime and nighttime conditions in a controlled weather environment.

3. The REHEARSE-3D Dataset

3.1. Background

The REHEARSE (adveRse wEatHEr datAset for sensoRy noiSe modEls) dataset [10] is introduced to facilitate the enhancement, comparison, and benchmarking of automotive perception sensor models under “controlled” and “fully characterized” severe weather conditions, including rain and fog. As a weather-aware dataset designed explicitly for autonomous driving systems, REHEARSE makes a valuable contribution to the research community. REHEARSE enables the rehearsing and comparing of sensor noise models in both simulated and real-world environments.

The REHEARSE dataset employs four automotive-grade sensors, including both passive and active automotive-grade sensors. Two distinct LiDAR technologies are incorporated: one featuring a rotating mechanical system (Ouster 128) and another utilizing micro-electromechanical systems (MEMSs). A 4D RADAR is included, which provides information on distance, velocity, azimuth, and elevation as its fourth dimension. The sensor suite is completed with an RGB camera and a FLIR thermal camera. Figure 2 illustrates the arrangement of the REHEARSE sensor suite.

Figure 2.

CARISSMA Outdoor test track (left) and REHEARSE sensor setup (right).

REHEARSE utilizes a set of mobile water sprinklers to generate synthetic rain while regulating the rain intensity in the CARISSMA outdoor test track (See Figure 2). Three rain intensities are measured: $10\mathrm{mm}/\mathrm{h}$ with a maximum span of $112\mathrm{m}$, $25\mathrm{mm}/\mathrm{h}$ with a maximum span of $56\mathrm{m}$, and $50\mathrm{mm}/\mathrm{h}$ with a maximum span of $28\mathrm{m}$. These values are obtained during both nighttime and daytime testing. REHEARSE has a range of target/sensor distances, varying from 10 to 112 m, and encompasses diverse weather conditions, including clear skies, heavy rainfall, and fog, with varying intensity levels. For additional technical details regarding the data logging setup, please refer to [10].

Additionally, REHEARSE offers critical parameters for describing weather phenomena, including rain intensity, droplet size distribution, wind velocity and direction, and visibility. This enhances the fidelity of sensor models. Weather effects, rain, and fog are validated using disdrometers, the square-meter method, weather stations, and visibility measurements. Weather conditions are evaluated using real rainfall data and the theoretical Marshall–Palmer droplet size distribution. This approach demonstrates that the generation of weather phenomena approximates real and theoretical counterparts.

Several sensors have been used on the REHEARSE [10] sensor suite. The MEMS LiDAR used has an angular resolution of 0.1° × 0.1° and a field of view (FOV) of 115° × 25°, scanning at 10 frames per second. Within the FOV, the maximum number of scanned lines is 256. For the RADAR point cloud, a high-resolution ZF ProWave sensor has been used. For RGB images, a LUCID Triton camera with a FOV of 80° was used, and for thermal imaging, a Forward-Looking Infra-Red (FLIR) camera with a FOV of 50° was used. In addition to these, there is a rotating 128-channel LiDAR sensor. However, we note that we only annotated the MEMS LiDAR and RADAR point clouds.

3.2. Data Annotation

REHEARSE-3D focuses on the clean and rainy REHEARSE data logged exclusively at the CARISSMA Outdoor facilities. REHEARSE-3D is the semantically enhanced iteration of the original REHEARSE dataset [10], briefly introduced in Section 3.1. To introduce the semantic information, we developed a customized automatic annotation framework to label MEMS LiDAR and 4D RADAR point clouds in REHEARSE [10].

Our data annotation process involves the following steps:

1. The unlabeled raw data is extracted from the dataset and stored in x, y, z, and intensity formats, resulting in 143 sequences containing about 300 dense LiDAR point clouds per sequence.

2. We then estimate the road plane using the RANSAC algorithm and keep it for later usage.

3. We create bounding boxes for the rain sprinklers and label them accordingly. Points that are below the estimated road plane are omitted.

4. Next, we create bounding boxes for the other objects on the road surface (e.g., pedestrians, bicyclists, cars, target objects) and label them accordingly.

5. A 2D polygon bounding box for the road is then created.

6. Any points above the road plane, within the 2D polygon bounding box, that do not belong to the previously labeled classes are then labeled as raindrops.

7. The remaining points on the 2D polygon are further labeled as a part of the road.

8. All the remaining points are labeled as background.

9. Since the object bounding boxes may vary slightly from sequence to sequence due to noise in the sensor readings, we manually correct the bounding boxes for each sequence and repeat the process from steps 2 to 8.

10. The corresponding 4D RADAR sparse point clouds are then automatically annotated by transferring labels from the nearest LiDAR points, utilizing a nearest-neighbor threshold of k = 5.

3.3. Data Statistics

There are, in total, 143 sequences annotated in REHEARSE-3D. As illustrated in Figure 3, $39\%$ of these sequences are obtained under clean conditions, whereas the remaining $61\%$ involve rain emulated at varying densities.

Figure 3.

REHEARSE-3D Data Distribution.

Figure 4 shows the intensity heat map of the synthetic rain generated during outdoor testing at CARISSMA. The rain intensity is measured in each square meter for a nominal rain intensity of $10\mathrm{mm}/\mathrm{h}$. In the heat map, the red region indicates high intensity, with values up to $14\mathrm{mm}/\mathrm{h}$, while the blue area corresponds to low intensity. As shown in Figure 4, the uniformity persists up to a distance of 80 m from the sensor ring. Within this range, the intensity fluctuates between 8 and $14\mathrm{mm}/\mathrm{h}$. Beyond a range of 80 m, a noticeable decline in intensity is observed, with values falling below $2\mathrm{mm}/\mathrm{h}$ and reaching insignificant intensity levels. The maximum rain accumulation is observed around (25, −2) and (58, 2), which refer to (X, Y) positions in meters. Although the average rain intensities align with the nominal values, the non-uniform intensity distribution is caused by wind, which is a result of outdoor testing. At a rain intensity of $10\mathrm{mm}/\mathrm{h}$, our sensor suite recorded an average wind speed of $4.14\mathrm{km}/\mathrm{h}$ with a standard deviation of $1.33\mathrm{km}/\mathrm{h}$. The average wind direction is 24.7°, exhibiting a high standard deviation of 27.4°. This significant variance is expected, reflecting the natural fluctuations in wind direction during precipitation events.

Figure 4.

Intensity heat map of the emulated rain at the CARISSMA outdoor test track—nominal intensity of $10\mathrm{mm}/\mathrm{h}$. Uniformity is influenced by wind.

In REHEARSE-3D, the annotation comprises point-wise annotations of eight semantic classes: sprinkler, car, pedestrian, bike, targets, road, rain, and background. Figure 5 depicts the comprehensive statistics of the point-wise annotations. It is evident from the figure that certain classes (such as background, road, and rain) predominate, while others, including pedestrians, bikes, and cars, exhibit a lower frequency of occurrence. It is noteworthy that this class imbalance problem is a pervasive issue in many existing datasets.

Figure 5.

Number of annotated points in each semantic class in REHEARSE-3D, separated by rain densities.

To the best of our knowledge, REHEARSE-3D is the first large-scale, fully point-wise annotated dataset to benchmark emulated rain data in 3D space. Table 1 provides a comparative analysis of REHEARSE-3D with several existing point cloud datasets.

3.4. Data Illustration

Figure 1 shows a sample of annotated LiDAR and 4D RADAR point cloud data with the corresponding RGB and thermal images. Note that these images are not annotated but are shown only for the sake of clarity.

Furthermore, Figure 6 shows examples of annotated 3D point cloud data captured in clean and rainy conditions in daytime and nighttime. We note that, though RADAR and LiDAR point clouds and thermal images are not affected by the time of the day, we provided daytime/nighttime information to facilitate research with corresponding RGB images.

Figure 6.

Annotated sample LiDAR point clouds (right) together with the corresponding RGB (left) and thermal camera images (middle) from nighttime—clean, daytime—rain, and nighttime—rain conditions (from top to bottom), respectively.

3.5. Rain Simulation

In addition to the rain-emulated sequences provided in REHEARSE-3D, we create corresponding rain-simulated counterparts for comparative evaluation. We adopt the physically accurate LiDAR rain model proposed by Espineira et al. [29] and apply it to clean point cloud sequences. Unlike Espineira et al.’s original implementation, which simulates each laser beam within a simulation engine (e.g., Unreal Engine), we apply the model to real-world recordings. To this end, we introduce a pre-processing step to retrieve each laser beam data point from recorded point clouds, enabling the identification of unreturned beams. We project the point clouds onto Polar Grid Maps (PGMs) using the known azimuth and elevation angles of the calibrated LiDAR sensor. Unreturned beams are represented as points with maximum range and zero intensity, and the PGMs are subsequently reshaped back into list-based point cloud format, as detailed in Algorithm 1. Following the same logic as Espineira et al.’s model, if a raindrop is detected within the maximum range of an unreturned beam, the resulting point is added to the point cloud; otherwise, the beam remains unreturned.

Raindrop diameters range from 0.5 mm to 6 mm, and the rain is simulated at intensities of 10, 25, and $50\mathrm{mm}/\mathrm{h}$, consistent with the rain-emulated settings in REHEARSE-3D. The resulting dataset is aligned with the original distribution of sequences, targets, and annotation classes to ensure a fair comparison.

We name this simulated dataset WMG simulated data as the simulation was performed by the Warwick Manufacturing Group (WMG) at the University of Warwick.

Algorithm 1 Pre-processing to retrieve unreturned LiDAR beams. Inputs: $\mathbf{X}\in {\mathbb{R}}^{N\times 3}$ (3D Cartesian coordinates), $\mathbf{I}\in {\mathbb{R}}^N$ (intensity values), ${\mathbf{\Theta }}_{\mathrm{calib}}\in {\mathbb{R}}^V$ and ${\mathbf{\Phi }}_{\mathrm{calib}}\in {\mathbb{R}}^H$ (calibrated elevation and azimuth angles), $R_{max}\in \mathbb{R}$ (maximum sensor range).

1: function PreProcess($\mathbf{X},\mathbf{I},{\mathbf{\Theta }}_{\mathrm{calib}},{\mathbf{\Phi }}_{\mathrm{calib}},R_{max}$)   2:    Init: ${\mathbf{X}}_{\mathrm{pgm}}\in {\mathbb{R}}^{V\times H\times 3},{\mathbf{I}}_{\mathrm{pgm}},{\mathbf{R}}_{\mathrm{pgm}}\in {\mathbb{R}}^{V\times H}$ as zeros   3:    $(\mathbf{R},\mathbf{\Phi },\mathbf{\Theta })\leftarrow \mathtt{ToPolar}\left(\mathbf{X}\right)$   4:    ${\mathbf{\Theta }}_{\mathrm{idx}}\leftarrow {\mathtt{argmin}}_i|\mathtt{sort}\left({\mathbf{\Theta }}_{\mathrm{calib}}\right)\left[i\right]-\mathbf{\Theta }|$   5:    ${\mathbf{\Phi }}_{\mathrm{idx}}\leftarrow {\mathtt{argmin}}_j|\mathtt{sort}\left({\mathbf{\Phi }}_{\mathrm{calib}}\right)\left[j\right]-\mathbf{\Phi }|$   6:    ${\mathbf{X}}_{\mathrm{pgm}}[{\mathbf{\Theta }}_{\mathrm{idx}},{\mathbf{\Phi }}_{\mathrm{idx}}]\leftarrow \mathbf{X}$;   ${\mathbf{I}}_{\mathrm{pgm}}[·]\leftarrow \mathbf{I}$;   ${\mathbf{R}}_{\mathrm{pgm}}[·]\leftarrow \mathbf{R}$   7:    $M\leftarrow ({\mathbf{R}}_{\mathrm{pgm}}==0)$;   ${\mathbf{R}}_{\mathrm{pgm}}\left[M\right]\leftarrow R_{max}$   8:    ${\mathbf{\Phi }}_{\mathrm{full}},{\mathbf{\Theta }}_{\mathrm{full}}\leftarrow \mathtt{meshgrid}({\mathbf{\Phi }}_{\mathrm{calib}},{\mathbf{\Theta }}_{\mathrm{calib}})$   9:    ${\mathbf{X}}^{\prime }\leftarrow \mathtt{ToEuclidean}({\mathbf{R}}_{\mathrm{pgm}},{\mathbf{\Phi }}_{\mathrm{full}},{\mathbf{\Theta }}_{\mathrm{full}})$ 10:    ${\mathbf{X}}_{\mathrm{pgm}}\left[M\right]\leftarrow {\mathbf{X}}^{\prime }\left[M\right]$ 11:    return ${\mathbf{X}}_{\mathrm{pgm}}.\mathtt{reshape}(-1,3),{\mathbf{I}}_{\mathrm{pgm}}.\mathtt{reshape}(-1)$ 12:end function

4. Benchmark on 3D Point Cloud De-Raining

In inclement weather conditions, the quality of the 3D point clouds captured by LiDAR and 4D RADAR sensors can be substantially degraded by precipitation (e.g., raindrops). The harsher the precipitation is, the noisier the point cloud becomes. As shown in recent works, such degradation can lead to substantial performance drops in downstream perception tasks, such as semantic segmentation [2,3] and object detection. This challenge has prompted extensive research on detecting and removing noisy point cloud data points. Such point cloud denoising (also known as de-raining) is a binary segmentation task where each raindrop is treated as an outlier to be filtered out.

Leveraging REHEARSE-3D, we explore detecting and removing emulated raindrops from early merged LiDAR and 4D RADAR point clouds. More specifically, we conduct an in-depth empirical study to evaluate the performance of several statistical and deep learning models on de-raining REHEARSE-3D point clouds, encompassing both clean and rainy cases in daytime and nighttime conditions, as illustrated in Figure 6.

4.1. Benchmark Setup

Baselines: We employ some unsupervised statistical filtering approaches. Radius Outlier Removal [30] (ROR) identifies a point as noise if it has fewer neighbors than a predefined threshold within a specified radius. Statistical Outlier Removal [30] (SOR) removes a point in case it lies further away from its neighbors than average. Dynamic ROR [31] (DROR) and Dynamic SOR [7] (DSOR) are dynamic extensions of ROR and SOR, respectively. DROR and DSOR are far superior to the ROR and SOR; hence, we only run the denoising with DROR and DSOR. DROR differs from ROR in that it computes a dynamic radius by reviewing the density of distant points [31]. In DSOR, a local distance threshold is dynamically computed based on data statistics [7]. It is worth noting that all these statistical methods are computationally intensive.

Additionally, we train three deep neural networks: SalsaNext [32], LiSnowNet-L1 [33], and 3D-OutDet [3]. SalsaNext [32] is a general-purpose semantic segmentation model trained to remove rain points as a binary-segmentation task. In SalsaNext [32], 3D point clouds are converted to 2D range view images to be further segmented by an advanced Convolutional encoder–decoder architecture. On the other hand, LiSnowNet [33] is introduced as a task-specific model for denoising point clouds in an unsupervised manner. LiSnowNet is specifically designed for point cloud data corrupted with snowfall. Therefore, it relies on the assumption that natural signals are sparse under the Fourier Form Transformation and the Discrete Wavelet Transformation [33]. However, when point clouds are corrupted with snowfall, the signal becomes less sparse under those transformations. The 3D-OutDet model [3] is a lightweight deep learning model that considers the local neighborhood relations to search for noisy points. These models were selected to represent distinct segmentation categories: SalsaNext as a supervised general-purpose segmenter, 3D-OutDet as a supervised point cloud denoiser, and LiSnowNet as an unsupervised denoiser.

Evaluation Metric: Following the existing literature [3,5], to assess the 3D point cloud de-raining performance, we employed $precision={\displaystyle \frac{TP}{TP+FP}}$, $recall={\displaystyle \frac{TP}{TP+FN}}$, $F_1$ score, and Intersection over Union (IoU) as quantitative metrics. Specifically, the $F_1$ score and IoU are defined as follows:

$$F_1=2\times {\displaystyle \frac{precision\times recall}{precision+recall}},$$ (1)

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}},$$ (2)

where $TP$ denotes True Positives as the correctly identified rain points, $FP$ refers to False Positives as the non-rain points that are misidentified as rain, and $FN$ indicates False Negatives as the instances where rain points were not detected correctly.

Implementation Details: We split REHEARSE-3D into three parts, including 25,507 scans of full 3D data for training, 8301 for validation, and 8602 for testing. The train and test splits are balanced with a similar proportion of point clouds from clean and rainy conditions. Like the WADS [7], SemanticSpray [4] or WeatherNet [5] datasets, our dataset also shows class imbalance since the number of points generated from the raindrops constitutes a small portion of the point cloud. As this case is prevalent in all the real-scanned datasets, all the baseline models employ techniques (e.g., class-weighted loss function) to fight this imbalance.

We follow the training protocol in [3] and report the model with the best performance. In brief, the training protocol adopts the default hyperparameters for SalsaNext and LiSnowNet as specified in their original publications. In contrast, 3D-OutDet was trained using a learning rate of $1\times {10}^{-2}$, a neighborhood size of 9, and a loss function combining weighted cross-entropy and Lovasz–Softmax loss [34]. The statistical filters are tuned using Optuna (https://optuna.org/, accessed on 20 January 2026). The parameter search is performed on a subset of 100 randomly generated samples for 100 iterations.

4.2. Benchmark Results

Quantitative Results: Table 2 shows the de-raining scores obtained on REHEARSE-3D for different state-of-the-art deep learning and statistical methods. Note that with the exception of SalsaNext [32] and 3D-OutDet [3], all methods are trained in an unsupervised manner.

Table 2.

Raindrop detection results on the REHEARSE-3D validation and test splits. All scores are in percentages. S and US stand for supervised and unsupervised training, respectively. ↓ denotes that lower is better, while ↑ indicates that higher is better. Computation time scores are taken from [3].

		Validation				Test
Model	Type	Precision ↑	Recall ↑	F1 ↑	IoU ↑	Precision ↑	Recall ↑	F1 ↑	IoU ↑	Time (ms) ↓
3D-OutDet [3]	Deep Learning—S	96.35	98.48	97.40	94.94	97.23	96.61	96.92	94.03	82
SalsaNext [32]	Deep Learning—S	95.17	99.41	97.24	94.61	95.81	99.07	97.41	94.92	97
LiSnowNet-L1 [33]	Deep Learning—US	14.42	31.94	19.55	12.44	21.22	37.70	26.07	16.86	131
DSOR [7]	Statistical—US	15.50	38.18	22.05	12.39	25.73	49.33	33.82	20.35	253
DROR [31]	Statistical—US	5.88	76.32	10.91	5.77	10.78	73.48	18.81	10.38	199

3D-OutDet [3] has the best precision and the lowest computational cost, while SalsaNext [32] has the best recall for both the validation and test sets. Again, 3D-OutDet [3] achieves the best $F_1$ score and IoU for the validation set, whereas SalsaNext [32] achieves the best $F_1$ score and IoU for the test set. However, we observe a significant performance drop across all unsupervised models. This clearly shows that a network trained with well-annotated supervision signals can accurately learn the underlying distribution of the emulated raindrops.

It is noteworthy that LiSnowNet-L1 [33] performs relatively poorly in detecting raindrops. We strongly believe that this is because LiSnowNet-L1 [33] is explicitly designed as a task-specific model to detect snowflakes only. The model is carefully designed to identify sparsity in point clouds subjected to specific transformations, such as the Fast Fourier or Discrete Wavelet Transformations. However, these hard constraints do not apply to raindrops, resulting in the lowest detection scores.

Table 3 shows the de-raining quantitative results for both WMG simulated data and the REHEARSE-3D emulated data. The experiments are divided into three categories based on rain intensity (heavy, medium, and light). In this experiment, we employ a supervised deep learning model (3D-OutDet [3]) and an unsupervised statistical model (DSOR [7]) for benchmarking. We observe that both models perform better on the WMG simulated data, and the deep learning-based 3D-OutDet [3] achieves nearly $100\%$ accuracy across all rain intensities. This is because the simulation is generated according to Algorithm 1, and 3D-OutDet [3] can easily capture the simulation process. Note that the same model performs less on the emulated REHEARSE-3D dataset, which is not mathematically generated. Even though the rain has been emulated using water sprinklers, environmental factors (e.g., wind direction, wind velocity, gravity, temperature) have shifted the emulated rain data distribution towards a more realistic distribution. As a result, both 3D-OutDet [3] and DSOR [7] face challenges in de-raining the emulated data in REHEARSE-3D.

Table 3.

Performance Comparison between simulated (WMG simulated) and emulated (REHEARSE-3D) data. ↓ denotes lower is better, while ↑ indicates higher is better.

		WMG Simulated				REHEARSE-3D
Model	Rain Density	Precision ↑	Recall ↑	F1 ↑	IoU ↑	Precision ↑	Recall ↑	F1 ↑	IoU ↑
3D-OutDet	Heavy	99.99	99.99	99.99	99.99	98.30	96.43	97.35	94.85
	Medium	99.99	99.99	99.99	99.99	97.88	95.89	96.87	93.93
	Light	99.99	99.99	99.99	99.99	98.08	98.67	98.38	96.81
DSOR	Heavy	78.58	99.59	87.85	78.33	27.08	39.46	32.12	19.13
	Medium	69.00	99.81	91.59	68.91	38.24	56.27	45.54	29.48
	Light	55.03	94.34	70.97	55.00	23.35	35.95	28.31	16.49

Computation Time: For the computation time, we report the total execution time provided in [3], which includes all pre-processing, post-processing, and runtime scores in milliseconds. As reported in [3], the runtime experiments were conducted on an Ubuntu 20.04 system featuring dual Intel ®Xeon(R) Silver 4210R CPUs (Intel Corporation, Santa Clara, CA, USA), two Nvidia RTX A6000 GPUs (NVIDIA Corporation, Santa Clara, CA, USA), and 192 GB of system memory.

In our experiments, SalsaNext [32] stands out as the best model for de-raining 3D point clouds, whereas 3D-OutDet [3] proves to be the most economical choice, offering the lowest latency.

Qualitative Results: Figure 7 depicts sample qualitative results for SalsaNext [32] and 3D-OutDet [3]. This figure clearly shows that these models can, to a great extent, detect and filter out raindrops while preserving other irrelevant points.

Figure 7.

(a) Original emulated rainy scene, (b) rain removed by SalsaNext [32], and (c) rain removed by 3D-OutDet [3].

5. Discussion

In this paper, we introduced REHEARSE-3D, a new multi-modal dataset for weather-aware autonomous driving research, offering a comprehensive collection of emulated rain data. As the first dataset of its kind, REHEARSE-3D features point-wise, semantically annotated, high-resolution LiDAR-256 scans, coupled with 4D RADAR point clouds, captured in both daytime and nighttime rainy conditions. Compared to other similar datasets, e.g., WADS [7], SemanticSpray [4], and WeatherNet [5], our proposed REHEARSE-3D has more annotated points across eight different semantic classes. As the scenes were static, no new classes were introduced after the scenes were set up. We also provide a plethora of information collected from other sensors, which makes REHEARSE-3D unique. Our experimental results show that deep learning-based models excel at raindrop detection, whereas the statistical models struggle with it (see Table 2). We also show that the gap between simulated and emulated data is quite large (see Table 3). Therefore, we argue that simulated data alone is insufficient for studying the nature of weather-generated point cloud noise.

Limitations: REHEARSE-3D involves only front-view static scenes in a controlled environment; therefore, neither the sensor ring nor objects in the scene (e.g., cars, pedestrians, etc.) are moving. As in the case of all other emulated rain data, there still exists a gap between our generated and real rain data. This gap, however, is less than that observed between the simulated and real data. Although RGB and thermal camera images, as well as Ouster LiDAR data, are available, we only annotated the MEMS LiDAR point clouds merged with 4D RADAR data.

6. Conclusions

REHEARSE-3D goes beyond typical datasets by providing detailed precipitation characteristics, including rain intensity, droplet size distribution, wind data, and visibility information. This wealth of data enables not only sensor noise modeling but also the point-level analysis of weather impacts on sensors.

We demonstrate the utility of our REHEARSE-3D dataset as a novel point cloud de-raining benchmark by comparing the performance of various statistical and deep learning models on REHEARSE-3D. As shown in Table 2, the supervised 3D-OutDet and SalsaNext models deliver robust performance, achieving 94.03% and 94.92% IoU, respectively. Conversely, the unsupervised models struggle significantly; DSOR reaches a peak IoU of only 20.35%. This performance gap highlights the utility of our dataset as a rigorous testbed for developing more effective unsupervised raindrop detection algorithms.

Author Contributions

A.M.R.: methodology, software, validation, formal analysis, investigation, data curation, visualization, writing—original draft preparation; J.H.: software, validation, visualization; H.H.: software, writing; Y.P.: data curation, visualization; M.F.D.: data curation, visualization; V.D.: conceptualization, supervision, writing—review and editing; E.E.A.: conceptualization, methodology, resources, supervision, writing—review and editing, project administration, funding acquisition. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The dataset and benchmark models are publicly available at https://sporsho.github.io/REHEARSE3D (accessed on 20 January 2026).

Conflicts of Interest

The authors declare no conflicts of interest.

Funding Statement

Co-funded by the European Union. The views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them. Project grant no. 101069576.