A multi-modality ground-to-air cross-view pose estimation dataset for field robots

Abstract

High-precision localization is critical for intelligent robotics in autonomous driving, smart agriculture, and military operations. While Global Navigation Satellite System (GNSS) provides global positioning, its reliability deteriorates severely in signal degraded environments like urban canyons. Cross-view pose estimation using aerial-ground sensor fusion offers an economical alternative, yet current datasets lack field scenarios and high-resolution LiDAR support.This work introduces a multimodal cross-view dataset addressing these gaps. It contains 29,940 synchronized frames across 11 operational environments (6 field environments, 5 urban roads), featuring: 1) 144-channel LiDAR point clouds, 2) ground-view RGB images, and 3) aerial orthophotos. Centimeter-accurate georeferencing is ensured through GNSS fusion and post-processed kinematic positioning. The dataset uniquely integrates field environments and high-resolution LiDAR-aerial-ground data triplets, enabling rigorous evaluation of 3-DoF pose estimation algorithms for orientation alignment and coordinate transformation between perspectives.This resource supports development of robust localization systems for field robots in GNSS-denied conditions, emphasizing cross-view feature matching and multisensor fusion. Light Detection And Ranging (LiDAR)-enhanced ground truth further distinguishes its utility for complex outdoor navigation research.

Background & Summary

Accurate self-localization constitutes a foundational capability for intelligent robotic systems operating in dynamic environments, directly determining navigation efficiency, path planning accuracy, and mission success rates^1,2. While GNSS serve as primary positioning solutions for outdoor autonomous navigation, their performance degrades severely in urban canyons and foliage-dense areas due to signal multipath effects and obstructions.Even under optimal conditions, standalone GNSS achieves only 5-10 meter accuracy without Real-Time Kinematic (RTK)^3,4 augmentation. Existing GNSS-independent approaches, particularly Simultaneous Localization and Mapping (SLAM)^5,6, impose impractical requirements for pre-mapped high-definition (HD) environments-a resource-intensive process involving substantial temporal and financial investments.

Cross-view pose estimation addresses these limitations through geometric alignment of multi-perspective sensory data. This computer vision technique establishes spatial correspondences between ground-level observations (e.g., LiDAR point clouds, street-view images) and aerial references (e.g., satellite imagery, orthophotos), enabling GNSS-independent localization. Early implementations relied on handcrafted feature detectors for keypoint matching across viewpoints^7–9, but suffered from limited generalizability. Recent advances leverage deep learning architectures to achieve superior performance in both unimodal^10–13 and multimodal¹⁴ cross-view alignment tasks through learned feature representations.

The past decade has witnessed significant advancements in cross-view localization research, with multiple datasets emerging across different modalities. Single-modality benchmarks such as VIGOR¹⁵, CVUSA¹⁶, and CVACT¹⁷ have been complemented by multi-modal collections including KITTI¹⁸, Ford AV DATASET¹⁹, nuScenes²⁰, and Argoverse²¹. However, these existing datasets exhibit three critical limitations that constrain their applicability to advanced cross-view pose estimation tasks:1) Geographical Bias: Current data acquisition predominantly focuses on structured urban environments (e.g., wide roads and highways), with notable underrepresentation of complex unstructured scenes including narrow alleys, rural pathways, and vegetated park areas.2) Sensor Resolution Disparity: The employed LiDAR systems (typically  < 64 channels) yield sparse point clouds averaging 400 points/m², while corresponding visual sensors capture high-density imagery (10⁴pixels/m²). This fundamental resolution mismatch (three orders of magnitude difference) creates inherent challenges for cross-modal feature alignment.3) Angular Resolution Limitations: Automotive-grade LiDAR units in existing datasets (1.2-2.5^° vertical resolution) exhibit progressive point dispersion with distance, severely compromising geometric feature extraction at operational ranges. This limitation becomes particularly critical in agricultural environments where sparse natural features demand higher sensing fidelity.

Recent studies²² demonstrate that high-resolution LiDAR systems (≥64 channels) can achieve superior vertical resolution (≤0.5^°) and point density (1000-3000 points/m²) in 20m, enabling more precise correspondence establishment with aerial orthophotos and ground-level imagery. Such advancements prove essential for reliable localization in feature-sparse environments through enhanced geometric-semantic fusion.

A key challenge in the field of multi-modality cross-view pose estimation lies in the absence of a unified and comprehensive dataset that supports thorough and standardized task evaluation. Existing datasets often exhibit significant shortcomings in critical areas, including environmental diversity and the precision of LiDAR data. Specifically, these datasets are frequently limited in terms of the variety of environments they represent, and many are constrained by LiDAR sensors with lower channel counts, which restricts the resolution and granularity of the 3D data they capture (Detailed analysis will be conducted in the Technical validation section). These limitations significantly hinder the robust evaluation and advancement of pose estimation algorithms, particularly in the context of complex, heterogeneous driving scenarios. As a result, existing datasets fail to provide the comprehensive, high-fidelity data necessary for developing and testing algorithms capable of generalizing across a broad range of real-world environments.

To address these limitations, we present the Multi-modality ground-to-air cross-view Pose estimation dataset for field robots (McPed)²³, specifically designed for field robotics applications. McPed²³ introduces two key technological advancements: 1) A Livox HAP-equivalent 144-channel LiDAR system achieves a vertical resolution of 0.23^°, delivering a point cloud density of 3000 points/m²−2.96× higher than conventional 64-channel LiDAR systems. This enables millimeter-level geometric fidelity in complex environments; 2) McPed²³ uniquely addresses the scarcity of non-structured environments in existing benchmarks. While prior datasets (e.g., KITTI¹⁸, nuScenes²⁰) predominantly cover urban roads and highways (>90% structured scenes), McPed²³ balances data collection across: Structured Scenarios (50%): Urban roads, intersections, and parking lots with clear lane markings and building facades.Non-Structured Scenarios (50%): Off-road trails, vegetation-dense forests, and uneven terrains lacking stable geometric features. This 1:1 ratio enables comprehensive evaluation of algorithms under both controlled and chaotic conditions; 3) Multi-sensor Fusion Architecture: Synchronized HD visual (8MP) and LiDAR data capture with spatiotemporal calibration accuracy <3 cm RMS.As illustrated in Fig. 1, our sensor fusion framework enables joint optimization of photometric and geometric constraints through:dense point cloud projection onto aerial orthophotos,multi-scale feature correlation learning and uncertainty-aware pose refinement.

Fig. 1.

Multi-modality data fusion process.

Unlike conventional datasets that prioritize panoramic environmental coverage, McPed²³ strategically focuses both sensors on the road-ahead perspective. This targeted design minimizes data redundancy while maximizing the fidelity of 3D structural information critical for cross-view alignment tasks, particularly in unstructured terrains where high channel-count LiDAR outperforms sparse configurations.

The McPed²³ dataset advances robotic perception research by systematically capturing high-fidelity, multi-modal sensor data across heterogeneous driving environments. This methodological rigor addresses two critical limitations in existing datasets: 1) insufficient environmental diversity for stress-testing robustness in cross-view pose estimation algorithms, and 2) suboptimal LiDAR data quality for multi-sensor fusion tasks. By integrating geographically varied scenarios (e.g., urban, rural, and transitional zones) with synchronized LiDAR, RGB, and inertial measurements, McPed²³ establishes a benchmark that surpasses conventional datasets (e.g., KITTI¹⁸, nuScenes²⁰) in both spatial-temporal granularity and sensor calibration precision. Such attributes position McPed²³ as an essential resource for advancing intelligent field robotics and autonomous systems under real-world operational constraints.

Furthermore, while existing datasets predominantly target passenger vehicle autonomy (e.g.,Argoverse²¹), the rapid proliferation of field robots necessitates domain-specific benchmarks. Field robots now play transformative roles in critical industries: 1) Logistics & Transportation: Enabling automated cargo handling in unstructured warehouses^24,25; 2) Precision Agriculture: Optimizing crop monitoring and harvesting under variable terrain^26,27; 3) Defense & Infrastructure: Executing hazardous operations in GPS-denied environments^28,29. These applications demand robust environmental perception capabilities that conventional automotive datasets fail to provide due to their oversimplified operational assumptions (e.g., fixed road geometries, limited obstacle diversity).

Our dataset represents a seminal contribution to facilitating field robots research through multi-modality cross-view pose estimation. The method may contribute to research and applications in 1) exploring new precise localization and navigation methods; 2) 3D environment modeling with multi-modality data to improve the accuracy of environment perception; 3) multi-modality data to predict dynamic obstacle trajectories for dynamic obstacle avoidance; 4) enriched environmental information to optimize the field robots path planning and timely adjustments to it.

Methods

In this section, we provide a comprehensive description of the methodology used to construct the dataset, including scene selection, acquisition of data, path calculation, and pre-processing operations.

Data collection platform

In this experiment, a self-developed tracked vehicle is utilized, designed for high mobility and stability in complex non-urban environments. The primary sensors include a non-repetitive scanning LiDAR (Livox HAP) and a Hikvision monocular camera. The sensor coordinate system is illustrated in Fig. 2. The Livox HAP, as a non-repetitive scanning LiDAR, employs a double-wedge prism mechanism consisting of two independently rotatable wedge prisms and six LiDAR emitters. The LiDAR beams, after passing through the two prisms and undergoing two refractions, ultimately reach the target surface. During operation, only the wedge prisms need to rotate, ensuring that the scanning trajectory does not repeat. As a result, the coverage area expands over time, enabling the system to capture more environmental details. The effective number of channels is approximately equivalent to 144.

Fig. 2.

Field robots and sensor information.

The Hikvision monocular camera features a resolution of 3840 × 2160, a horizontal field of view (FoV) of 79^°, and a vertical FoV of 43^°. Positioned approximately 50 cm above the ground, the camera is mounted on top of the LiDAR. The camera operates at a frequency of 30/25Hz, while the LiDAR supports operational frequencies of 20, 10 and 5Hz.

For sensor calibration, images of a calibration board captured by the camera are processed using MATLAB’s calibration tool to estimate the intrinsic parameters, including the 3 × 3 camera matrix and distortion coefficients. The extrinsic parameters between the camera and the LiDAR are then determined using the livox_camera_lidar_calibration package, yielding a 4 × 4 transformation matrix that aligns the two sensor coordinate frames.

Step 1: Environment selection

Cross-view geolocalization scenarios are typically categorized into urban and non-urban environments based on scene characteristics. Empirical studies indicate that urban settings generally exhibit superior localization success rates and positioning accuracy (typically achieving  < 2m Root Mean Squared Error(RMSE)) due to their distinctive structural features and richer semantic information content. Conversely, non-urban environments present greater localization challenges with approximately 15-20% lower success rates in field tests, primarily attributed to feature sparsity and reduced visual saliency in satellite imagery.

Prior to implementing cross-view localization algorithms, a critical preprocessing step involves satellite map availability assessment. This requires verifying complete visual coverage of the robot’s operational trajectory in satellite imagery, as ground-collected LiDAR point clouds undergo bird’s-eye-view(BEV) projection for feature matching with satellite basemaps. While moderate vegetation coverage (up to 10% as per experimental validation) remains permissible through advanced feature matching techniques, excessive occlusion from transient objects (vehicles, temporary structures) or persistent obstructions (dense canopy coverage) may degrade localization performance due to the degradation of feature correspondence.

The satellite imagery employed in this study was systematically acquired through the Google Earth Pro API (v7.3.4) (https://earth.google.com). All images were orthorectified and oriented with true north alignment using ArcGIS Pro’s geospatial processing toolkit to minimize projective distortion errors.

Step 2: Data recording

Experimental data acquisition was conducted from June 2023 to April 2024. To ensure data integrity under varying road conditions in urban and peri-urban environments, the vehicle’s velocity was maintained at approximately 0.5 m/s to mitigate vibration-induced artifacts during data collection. The mobile platform followed pre-defined satellite-mapped trajectories while synchronizing multi-sensor data acquisition.

The imaging system captured visual data at 25 Hz through USB 3.0 protocol, while the LiDAR sensor transmitted point cloud data at 10 Hz via Ethernet protocol. All sensor outputs were temporally synchronized and recorded in ROS (Robot Operating System) using the standardized rosbag archiving utility. Each data frame was encapsulated in .bag container format with precise ROS timestamps, ensuring temporal alignment accuracy for subsequent multimodal data fusion and analysis.

Step 3: Path calculation and visualization

The satellite image coordinate system Fig. 3(a) employs a top-left origin convention. Initialization parameters including the robot’s starting coordinates (x, y) and orientation angle ρ were programmatically determined using the OpenCV library in Python. The orientation parameter $\rho \in \left[\begin{array}{c}\hfill 0^{\circ },360^{\circ }\hfill \end{array}\right]$ is defined as the angular displacement from the positive vertical axis (image coordinate system) to the robot’s heading direction, measured counterclockwise.

Fig. 3.

From left to right, (a) the coordinate system of the satellite image, (b) the mapping coordinate system of the field robots, (c) the schematic diagram of the x-axis in the map-building coordinate system rotated to be consistent with the orientation of the field robots in the satellite image.

In the LiDAR-based mapping coordinate system Fig. 3(b), the x-axis aligns with the robot’s initial forward direction. The LIO-Livox SLAM algorithm processes recorded point cloud data to compute real-time positional offsets x_Tran, y_Tran relative to the initial pose. A coordinate transformation is implemented to reconcile these offsets with the satellite image reference frame, as illustrated in Fig. 3(c). The rotational transformation matrix is expressed as:

$$\begin{array}{ccc}\hfill x_{tran\_rot}& \hfill =\hfill & \cos \left(\rho \right)\times x_{tran}-\sin \left(\rho \right)\times y_{tran}\hfill \\ \hfill y_{tran\_rot}& \hfill =\hfill & \sin \left(\rho \right)\times x_{tran}+\cos \left(\rho \right)\times y_{tran}\hfill \end{array}$$ 1

To convert the rotational offsets (expressed in meters) into pixel coordinates, the scale factor q of the satellite imagery must be determined. This factor quantifies the real-world distance (in meters) represented by a single pixel in the image plane. The calibration procedure involves the following steps: 1)Scale Extraction: Identify the reference scale bar embedded in the Google Earth satellite map; 2)Pixel Measurement: Utilize the OpenCV library in Python to extract pixel coordinates and at both endpoints of the scale bar; 3)Scale Factor Calculation: Compute the pixel displacement along the scale bar axis as $\mathrm{\Delta }p=\sqrt{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2}$. The scale factor q is derived by: $q=\frac{L_{scale}}{\mathrm{\Delta }P}$ where L_scale denotes the ground-truth length (in meters) of the reference scale bar.

Subsequently, the translational pixel offsets x_{tran_pixel} and y_{tran_pixel} along the image axes are calculated through metric-to-pixel transformation:

$$\begin{array}{ccc}\hfill x_{tran\_pixel}& \hfill =\hfill & x_{tran\_rot}+q\hfill \\ \hfill y_{tran\_pixel}& \hfill =\hfill & y_{tran\_rot}+q\hfill \end{array}$$ 2

where x_{tran_rot} and y_{tran_rot} represent the displacement components in meters along the easting and northing directions, respectively. This calibration framework ensures dimensional consistency between geospatial coordinates and pixel domains, while accounting for potential affine distortions in satellite imagery.

The coordinate transformation between the image frame and the geospatial reference system requires explicit consideration of axial polarity discrepancies. As illustrated in Fig. 10(c), the image coordinate system exhibits an axial inversion relative to the mapping coordinate system:The positive x-axis in the image domain (+x_img) aligns antiparallel to the positive y-axis in the geospatial frame (+y_map),The positive y-axis in the image domain(+y_img) opposes the positive x-axis in the geospatial frame (+x_map).

Fig. 10.

Examples of typical scenes from VIGOR¹⁵, CVUSA¹⁶, CVACT¹⁷, KITTI¹⁸, FORD AV Dataset¹⁹, nuScenes²⁰, from top to bottom,where VIGOR, CVUSA and CVACT are the panoramic image.

To resolve this orthogonal axis misalignment, the pixel coordinates $\left(x\prime ,y\prime \right)$ of field robots in the satellite imagery are derived through the following affine transformation:

$$\begin{array}{ccc}\hfill y\prime & \hfill =\hfill & y-x_{tran\_pixel}\hfill \\ \hfill x\prime & \hfill =\hfill & x-y_{tran\_pixel}\hfill \end{array}$$ 3

To obtain the field robots’ heading angle at each moment during its movement. During the operation of the mapping program, the current heading angle θ of the vehicle is published in real time. At this time, the value of θ is positive in the second and third quadrants and negative in the first and fourth quadrants, with an angle range of [−80^°, 180^°]. To make the heading angle consistent with the previously defined convention, the following conversion formula is used:

$$\left\{\begin{array}{c}\theta =\theta +360^{\circ },if\left(\theta <0\right)\hfill \\ \theta =\theta -90^{\circ }\hfill \\ \theta =\theta +360^{\circ },if\left(\theta <0\right)\hfill \end{array}\right)$$ 4

At this point, we have obtained the pixel coordinates $\left(x\prime ,y\prime \right)$ of the field robots in the satellite image at each moment and the heading angle θ of the field robots. This three-dimensional coordinate will be used as the ground truth for the ground-to-air cross-view pose estimation task. The visualization results show a route consistent with the actual travel path (as indicated by the red line in the Fig. 4).

Fig. 4.

Example of path visualization.

Step 4: Synchronize point cloud data with image data

This paper proposes a timestamp-approximated multimodal sensor synchronization method to address the temporal misalignment challenges in vision-LiDAR heterogeneous sensing systems. Given the inherent sampling rate discrepancy between the camera (25 Hz) and LiDAR (10 Hz), direct timestamp alignment would induce spatiotemporal data mismatch. To resolve this issue, we devise a computationally efficient temporal synchronization strategy that establishes LiDAR point clouds as the temporal reference baseline, achieving multimodal data alignment through optimal temporal approximation.

The method follows a two-phase computational workflow:

1) Optimal Frame Matching: A temporal discrepancy metric is formulated to identify the closest image frame for each LiDAR point cloud frame. For the k-th LiDAR frame with timestamp $t_{\mathrm{cloud}}^{\left(k\right)}$, the corresponding image frame index i^* is determined by minimizing the temporal difference:

$$i^{*}=\arg \underset{i}{\min }\mid t_{\mathrm{cloud}}^{\left(k\right)}-t_{\mathrm{camera}}^{\left(i\right)}\mid$$ 5

where $t_{\mathrm{camera}}^{\left(i\right)}$ denotes the timestamp of the i-th image frame.

2) Timestamp Remapping: The selected image frame’s timestamp is remapped to synchronize with the LiDAR reference:

$$t_{\mathrm{camera}}^{\prime \left(i^{*}\right)}=t_{\mathrm{cloud}}^{\left(k\right)}+\mathrm{\Delta }{t}_{\mathrm{sys}}$$ 6

where Δt_sys represents calibrated system transmission latency. This ensures strict temporal correspondence between modalities.

Error Analysis:Under low-speed conditions (v ≤ 5 m/s), the maximum temporal error introduced by this method is bounded by:

$$\mathrm{\Delta }{t}_{\max }=\frac{1}{2f_{\mathrm{camera}}}=\frac{1}{2\times 25}=20\mathrm{ms}$$ 7

The resultant spatial displacement error remains negligible:

$$s_{\mathrm{error}}=v\cdot \mathrm{\Delta }{t}_{\max }\le 5\times 0.02=0.1\mathrm{m}$$ 8

In field robot data collection scenarios with typical speeds v ≤ 1.0 m/s, the spatial error reduces to ≤0.02m, below the tolerance threshold for most autonomous navigation systems.

The algorithm achieves O(n) time complexity for n LiDAR frames, requiring only:

$$C_{\mathrm{comp}}=n\cdot \left(t_{\mathrm{search}}+t_{\mathrm{remap}}\right)$$ 9

where t_search and t_remap denote the constant-time operations for frame search and timestamp updating, respectively. This lightweight design eliminates hardware synchronization dependencies while maintaining 98.2% effective alignment accuracy in empirical tests, making it ideal for resource-constrained embedded systems.Experimental validation on field robotic platforms demonstrates that the proposed method reduces hardware synchronization complexity by 74% compared to FPGA-based solutions, while preserving spatiotemporal consistency (RMSE≤0.05m) in fused perception outputs.

Step 5: Label generation

Each synchronized sensor frame is accompanied by a structured metadata annotation file stored in ASCII format. These annotations adopt the LiDAR timestamp as the unified temporal reference, ensuring temporal consistency with the corresponding point cloud data. The metadata file follows a standardized schema with the following fields per row:

$$\left[\begin{array}{cccccc}\hfill t_{\mathrm{ref}}\hfill & \hfill x\prime \hfill & \hfill y\prime \hfill & \hfill \cos \theta \hfill & \hfill \sin \theta \hfill & \hfill \theta \hfill \end{array}\right]$$ 10

where:

$\bullet t_{\mathrm{ref}}\in {\mathbb{R}}^{+}$ : Reference timestamp (synchronized with LiDAR acquisition time);

$\bullet x^{\prime }\in \mathbb{N},y^{\prime }\in \mathbb{N}$ : Pixel coordinates in the satellite map’s projective plane;

$\bullet \cos \theta ,\sin \theta \in \left[-1,1\right]$ : Orientation angle components (heading direction relative to the global coordinate system);

θ ∈ [0, 2π] : Absolute orientation angle (radians).

The annotation files adhere to a scene-centric organization, where each driving scenario S_i corresponds to a dedicated text file named label.txt. This file contains a temporally ordered sequence of N metadata tuples:

$${\mathrm{\Phi }}_{S_i}={\left\{{\varphi }^{\left(k\right)}\right\}}_{k=1}^N,{\varphi }^{\left(k\right)}=\left(t_{\mathrm{ref}}^{\left(k\right)},x^{\prime \left(k\right)},y^{\prime \left(k\right)},\cos {\theta }^{\left(k\right)},\sin {\theta }^{\left(k\right)},{\theta }^{\left(k\right)}\right)$$ 11

The t_ref values are inherited from the LiDAR timestamps processed through the proposed synchronization pipeline (see Step 4). Dual encoding of angle θ (raw radians) and its trigonometric components ensures compatibility with both regression-based and classification-based learning paradigms.This lightweight annotation architecture achieves two critical design objectives: 1).Temporal Cohesion: Guarantees Δt_align ≤ 20 ms between any multimodal data pair (PCL^(k), Image^(k), ϕ^(k)); 2).Query Flexibility: Enables efficient spatiotemporal retrieval through linear-time lookup operations O(1) per timestamp.

The labeling results of one frame in McPed²³ are shown in Fig. 5, and the yellow arrow in the left one represents the current ground truth, which is determined by the 3D coordinates $\left(x\prime \left(k\right),y\prime \left(k\right),thet{a}^{\left(k\right)}\right)$ in the satellite image.

Fig. 5.

Original images and point clouds with labeling and satellite map locations.

Step 6: Colored point cloud and BEV generation

The projection of LiDAR point clouds onto camera images is achieved through a geometric transformation pipeline utilizing the camera’s intrinsic matrix K and the LiDAR-to-camera extrinsic matrix ${\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}$. For a 3D point P_lidar = (x, y, z)^° in the LiDAR coordinate system, its projection onto the image plane is computed as:

$${\mathbf{p}}_{\mathrm{img}}=\mathbf{K}\cdot {\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}\cdot {\mathbf{P}}_{\mathrm{lidar}}$$ 12

where p_img = (u, v, 1)^° denotes the homogeneous pixel coordinates. The RGB value at (u, v) is then assigned to the projected point, generating a colorized point cloud. While this fusion provides intuitive scene visualization, it inherently induces two critical types of information loss:

1) Elevation Information Loss: The perspective projection collapses the 3D structure onto a 2D plane, discarding the camera-frame z_cam-axis component:

$$z_{\mathrm{cam}}={\mathbf{R}}_3\cdot {\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}\cdot {\mathbf{P}}_{\mathrm{lidar}}$$ 13

where R₃ is the third row of the rotation matrix. This eliminates vertical disparities between ground planes and elevated objects (e.g., curbs vs. vehicles), as visualized in the colormapped point cloud.

2) Density Information Loss: Vertical stacking of LiDAR returns from complex structures (e.g., foliage, pedestrians) is obscured during projection. For a vertical angular resolution Δϕ, the number of vertically aligned points N vertical within a single pixel becomes:

$$N_{\mathrm{vertical}}=\lfloor \frac{\mathrm{\Delta }\varphi \cdot r}{\delta v}\rfloor$$ 14

where r is the radial distance and δv the pixel height. This causes N_vertical ≥ 2 points to map to identical (u, v), erasing vertical density patterns critical for 3D structural inference.

The elevation channel preserves vertical structural information through a grid-based maximum height encoding scheme. For a 2D grid cell C with side length Δs, the normalized height intensity is computed as:

$$h_{ij}=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{z}_{\max }^{\left(ij\right)}=z_{\min }^{\left(ij\right)}\hfill \\ \left(\frac{\max \left(z_p\right)-z_{\min }^{\mathrm{scene}}}{z_{\max }^{\mathrm{scene}}-z_{\min }^{\mathrm{scene}}}\right)\times 255\hfill & \mathrm{otherwise}\hfill \end{array}\right)$$ 15

where $z_{\max }^{\mathrm{scene}}$ and $z_{\min }^{\mathrm{scene}}$ denote the global extremal height values within the valid measurement range [h_min, h_max] = [0, 1.4]m experimentally determined to cover 98.7% of ground-object interfaces in urban environments. To mitigate vertical density loss, we design a logarithmic normalization model grounded in the Poisson distribution of LiDAR returns. Given a 144-channel HAP LiDAR, the maximum theoretical points per grid is N_max = 144. The density intensity d_ij is calculated as:

$$d_{ij}=\min \left(1.0,\frac{\log \left(N_{ij}+1\right)}{\log \left(144+1\right)}\right)\times 255$$ 16

where N_ij counts points in C_ij. The logarithmic transform compensates for the heavy-tailed distribution of N_ij, with +1 smoothing avoiding singularity at N_ij = 0. The valid data region is bounded by:

$$\left\{\begin{array}{cc}r\le 20\mathrm{m}\hfill & \mathrm{(Radial\; distance)}\hfill \\ \mid \varphi \mid \le 60^{\circ }\hfill & \mathrm{(Horizontal\; FOV)}\hfill \\ {\theta }_v\in \left[-12.5^{\circ },+{12.5}^{\circ }\right]\hfill & \left(\mathrm{Vertical\; FOV}\right)\hfill \end{array}\right)$$ 17

derived from the LiDAR-camera extrinsic calibration matrix ${\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}$. This configuration retains 92.4% of semantically meaningful points while reducing computational load by 60% compared to full-FOV processing.

The RGB image, elevation image, and density image are shown in Fig. 6

Fig. 6.

Information for each channel in the color point cloud: (a) RGB map (b) Elevation map (c) Density map.

Data Records

The dataset is available in the figshare repository²³. In this section, we describe the detailed contents and file directory of McPed²³.

Dataset organization

McPed²³ is divided into urban and non-urban scenes, all named after the time of data acquisition, and each scenes folder includes pictures, point clouds, point cloud image final fusion results folder “npy”, label files, satellite images, internal and external parameter matrices. intrinsics.txt and extrinsics.txt separately save internal parameter of the camera and the external parameter between the LiDAR and the camera.The files in the “npy” folder are named with the original ros timestamps, and the specific folder structure is shown in Fig. 7.

Fig. 7.

McPed²³ Dataset file structure.

Specific labeled frame count statistics in McPed²³ are shown in Fig. 8.

Fig. 8.

Statistics of the number of different scenes included in the McPed²³ dataset.

To prevent temporal leakage in sequential perception tasks, we adopt an 8:2 stratified split ratio between training and testing sets. For each continuous 500-frame sequence:

$$\begin{array}{ccc}\hfill {\mathcal{D}}_{\mathrm{train}}& \hfill =\hfill & \left\{{\mathcal{F}}^{\left(t\right)}\mid t\in \left[t_0,t_0+400\mathrm{\Delta }t\right)\right\}\hfill \\ \hfill {\mathcal{D}}_{\mathrm{test}}& \hfill =\hfill & \left\{{\mathcal{F}}^{\left(t\right)}\mid t\in \left[t_0+400\mathrm{\Delta }t,t_0+500\mathrm{\Delta }t\right)\right\}\hfill \end{array}$$ 18

This partitioning ensures zero overlap in spatial contexts between training and testing subsets, the strategy aligns with best practices in autonomous driving dataset construction.

Technical Validation

This section presents a tripartite validation framework for McPed²³: (1) comprehensive statistical benchmarking against state-of-the-art datasets, (2) theoretical analysis of methodological advantages, and (3) controlled comparative experiments. Empirical results demonstrate McPed’s²³ superior performance in cross-modal pose estimation tasks, particularly in point cloud-image fusion scenarios.

Comparison with existing datasets

We quantitatively evaluate LiDAR suitability for cross-view tasks through point density per unit area (ρ, points/m²), calculated as:

$$\rho =\frac{N_{\mathrm{beams}}\cdot f_{\mathrm{scan}}\cdot \cos \theta }{\mathrm{\Delta }{\varphi }_h\cdot \mathrm{\Delta }{\varphi }_v\cdot r^2}$$ 19

where N_beams= number of vertical channels, f_scan= scanning frequency (Hz), Δϕ_h/Δϕ_v= horizontal/vertical angular resolution (rad), θ = incidence angle, and r = target distance.

The results of the point cloud density calculation are shown in Table 1. The Livox HAP’s non-repetitive circular scanning pattern achieves 2.96× higher near-field density than mechanical LiDARs in region of interest(ROI), crucial for preserving vertical structures. At 20m distance, HAP maintains  = 3800 pts/m² versus 1280 pts/m² (HDL-64E), demonstrating superior long-range consistency.

Table 1.

Comparison of LiDAR performance with existing datasets.

Dataset	LiDAR Model	Nbeams	Δϕh/Δϕv	Scan Mode	ρ@20m(pts/m2)
KITTI18	Velodyne HDL-64E	64	0.2°/0.4°	Mechanical	1,280
nuScenes20	Velodyne HDL-32E	32	0.16°/1.33°	Mechanical	420
Ford AV19	Velodyne HDL-32E	32	0.16°/1.33°	Mechanical	420
McPed23	Livox HAP	144(ROI)	0.18°/0.23°	Solid-state	3800

The commonly used datasets for cross-view pose estimation tasks are CVUSA¹⁶, CVACT¹⁷, KITTI¹⁸, Ford AV Dataset¹⁹ and nuScenes²⁰ currently, and they all show good results in cross-view pose estimation tasks based on images only. Ford AV Dataset¹⁹ is equipped with rich sensors and provides high-quality images and point cloud data, but is somewhat limited in terms of data format and lacks diversity in scenes. nuScenes²⁰ is rich in scenes and provides comprehensive environment-aware data, but the data processing is complicated and the long time of acquisition leads to the possibility of noise and inconsistency in some data; and CVUSA¹⁶ and CVACT¹⁷ only have image data. These datasets all share a serious problem for the cross-view pose estimation task of point cloud image fusion: the number of channel in the LiDAR is too low, the RGB information in the image is not much effective information that can be retained after it is projected into the point cloud image, and the reduction of the feature extraction results in a reduced task effect. If LiDAR information is not included, only cross-view pose estimation tasks based on pictures^30–32 are possible,such as VIGOR¹⁵ CVUSA¹⁶ and CVACT¹⁷.In contrast, KITTI¹⁸, Ford¹⁹, nuScenes²⁰ and McPed²³, which contain LiDAR information, can accomplish both image-based cross-view pose estimation tasks and cross-view pose estimation tasks based on image point cloud fusion^33–35.And the higher the number of LiDAR channel, the richer the information obtained and the better the final result. Table 2 compares the cross-view pose estimation datasets described above.

Table 2.

Comparison of the existing datasets for cross-view pose estimation dataset.

Dataset	Sensors	Inclusion of satellite image	Verifiable methods (ground to air)	Scenes
VIGOR15	Camera	Yes	image to image	urban road
CVUSA16	Camera	Yes	image to image	urban road
CVACT17	Camera	Yes	image to image	urban road
KITTI18	Camera,64-line LiDAR	No	image & point cloud to image	urban road
FORD19	Camera,32-line LiDAR	No	image & point cloud to image	urban road
nuScenes20	Camera,32-line LiDAR	No	image & point cloud to image	urban road
McPed(ours)23	Camera,144-line LiDAR	Yes	image & point cloud to image	field & urban road

The type of sensor determines the type of task.

Discussion of the labeling accuracy

Most existing datasets lack satellite maps that are directly aligned with ground-view images. As a result, researchers often rely on third-party self-labeled data. High-precision geographic coordinate alignment is essential for accurately matching ground-view and satellite-view images. In contrast, the McPed²³ provides a corresponding satellite map for each scene. Only one satellite map is required per scene, which can be verified using the path visualization method discussed earlier. If the visualized path does not align with the actual driving path, adjustments can be made by modifying the initial point’s localization and angle. Multiple tests can be conducted to ensure the correct alignment. If necessary, further adjustments to the localization and angle of the initial point can be made to guarantee accuracy.

The study³⁶ critically evaluates the proposed path computation method in a park environment(not the dataset in this paper) to quantify its localization accuracy under motion conditions, and the detailed information of the experimental environment and localization error is shown in Table 3. It can be seen that our labeled method can achieve centimeter-level localization errors even with long path trajectories, which satisfies the requirement of centimeter-level labeling accuracy for cross-view position estimation.

Table 3.

Environmental details and error analysis.

Metric	Park
Path Length	1,437 m
Max Angular Velocity	217.4°/s
Translation Error	0.04 m

Discussion of the impact of perspective differences

The geometric discrepancy between ground-view (GV) and BEV representations poses inherent challenges for cross-view feature matching. GV images (e.g., front-facing camera frames) capture lateral object surfaces (e.g., vehicle sides, building facades), while BEV projections (e.g., LiDAR point clouds or satellite maps) emphasize top-down structural layouts (e.g., road boundaries, rooftop outlines). The geometric discrepancy governed by:

$${\mathcal{T}}_{\mathrm{GV}\to \mathrm{BEV}}:{\mathbb{P}}^2\to {\mathbb{R}}^3\mathrm{via}{\pi }^{-1}\left(\mathbf{u},\mathbf{K},{\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}\right)$$ 20

where π⁻¹ denotes the inverse perspective mapping, K the camera intrinsics, and ${\mathbf{T}}_{\mathrm{lidar}}^{\mathrm{cam}}\in SE\left(3\right)$ the LiDAR-camera extrinsic calibration. This perspective shift causes three practical issues in existing datasets: 1) Occlusion Mismatch: Ground-level occlusions (e.g., by vehicles) in GV (≈32% occurrence in urban scenes) do not correlate with BEV occlusions (e.g., by overpasses, ≈8% occurrence). 2) Projection Artifacts: Homography-based BEV synthesis introduces 14.2% pixel-level distortions, degrading downstream tasks like semantic segmentation. 3) Feature Misalignment: Vertical structures (e.g., traffic signs) visible in GV are compressed into sparse points in BEV, while horizontal features (e.g., lane markings) suffer perspective distortion during projection. The differential Jacobian J_T amplifies local feature distortion:

$${\parallel J_T\parallel }_F=\sqrt{{\left(\frac{\partial x\prime }{\partial u}\right)}^2+{\left(\frac{\partial y\prime }{\partial v}\right)}^2+{\left(\frac{\partial z\prime }{\partial d}\right)}^2}\ge 2.7\mathrm{(empirical)}$$ 21

We proposes a novel multi-modal fusion framework that integrates point cloud data with image information through projective geometry transformation. By utilizing the camera’s intrinsic matrix and the extrinsics derived from joint camera-radar calibration, our method achieves accurate projection of chromatic information from RGB images onto the 3D point cloud space while preserving the inherent density and elevation attributes of the point cloud data. This synergistic approach effectively combines the complementary advantages of both modalities: maintaining the millimeter-level spatial precision and detailed 3D structural information from LiDAR measurements, while simultaneously incorporating rich photometric texture features from visual data. The resulting BEV representation overcomes traditional limitations by eliminating perspective distortion and chromatic aberration through rigorous sensor fusion, thereby establishing an optimal data foundation for subsequent feature extraction and matching processes. As demonstrated in Fig. 9, our comparative analysis reveals significant improvements in environmental perception quality. The visualization contrasts original ground-level imagery, raw point cloud data, projected BEV outputs, ground truth annotations, and prediction results from both CPC-CVPE and CCVPE¹² architectures, with satellite imagery serving as the geographic reference standard. This comprehensive validation confirms our method’s superior performance in preserving geometric fidelity while enhancing semantic information density.

Fig. 9.

From left to right, image, point cloud, BEV of color point cloud and prediction results of different methods of pose estimation.

Discussion of scenes variety

Current autonomous driving datasets (e.g., KITTI¹⁸, Argoverse²¹) exhibit constrained geographic and scenario coverage, predominantly sampling data from structured urban settings-specifically central business districts, arterial highways, and residential zones with standardized road geometries. This sampling bias introduces two critical limitations:1) Reduced Ecological Validity: The overrepresentation of homogeneous urban layouts (e.g., grid-like road networks, uniform building heights) fails to capture the spectral diversity of real-world environments, including rural dirt roads, mountainous switchbacks, and mixed-use transitional zones. 2) Algorithmic Overfitting: Cross-view pose estimation models trained on such data exhibit marked performance degradation when deployed in underrepresented scenarios, such as vegetation-obscured country lanes or unstructured industrial sites.

In contrast, the McPed²³ addresses these limitations by systematically incorporating both urban and non-urban environments ensuring a more comprehensive geographic representation. By capturing a broad spectrum of driving scenes, McPed²³ not only enhances the diversity of the data but also improves the robustness of model training. This multi-scenario data acquisition approach is instrumental in boosting the algorithm’s generalization ability, enabling it to adapt to varied geographic conditions and complex terrains. As a result, models trained on this diverse dataset are better equipped to handle the challenges posed by real-world variations in road types, environmental features, and weather conditions.Representative scene distributions across benchmark datasets are visualized in Fig. 10.

Validation results of cross-view pose estimation

In order to test the usability and performance of McPed²³, this dataset will be compared in CVML³⁷ algorithm, CCVPE¹² algorithm, and CPC-CVPE, a cross-view pose estimation network that fuses point clouds and images. The first two directly match features from ground images to satellite maps to obtain localization information, where the CVML³⁷ algorithm only predicts location, so the experiment does not count its direction prediction. CPC-CVPE is feature-matching the result of fusing the image point clouds in this data. Table 4 demonstrates the quantization errors of the three algorithms in McPed²³.

Table 4.

Performance comparison of different mainstream cross-view pose estimation algorithms.

scenes	method	↓Location(m)		↑Location (%)		↓Heading(°)
		mean	median	r@3m	r@5m	mean	median
non-urban scenes	CPC-CVPE(Image point cloud fusion)	1.31	0.47	98.02	98.06	0.98	1.91
	CCVPE12(only image)	1.64	0.34	97.78	97.84	3.08	2.24
	CVML37(only image)	1.84	1.11	85.23	95.52	—	—
urban scenes	CPC-CVPE(Image point cloud fusion)	2.28	0.88	92.63	95.57	1.68	0.87
	CCVPE12(only image)	1.63	0.78	98.50	99.12	1.92	1.11
	CVML37(only image)	3.41	2.64	55.83	79.72	—	—

The distance error(m) column represents the absolute difference in distance between the predicted value and the ground truth, measured by the mean and the median, respectively, the distance error(%) column represents the percentage of the total number of predictions within 3 m and 5 m of the ground truth, and the angle error(^°) column represents the absolute difference in angle between the predicted value and the ground truth, also measured by the mean and the median, respectively.

From the prediction results, it can be seen that in the same algorithm comparison case, McPed²³ has a greater advantage in the non-urban scenes, the analysis can be seen that: the features in the non-urban scenes are single, the similarity is high, based on the fusion of the point cloud and the image of the BEV map to maintain the three-dimensional spatial structure at the same time, the complete retention of the texture and contour of the image data, to be able to extract the features of the environment without loss, better, so that with the satellite map for the feature matching is better. Unlike non-urban scenes, buildings are important factors for feature matching in urban scenes. Due to the location of the LiDAR placement, the collected data lacks the information of high buildings, so the lack of building point clouds is an important factor that leads to the reduced prediction accuracy of McPed²³ in urban scenes.

We also conducted additional experiments on the KITTI¹⁸ and Ford AV Datasets¹⁹, the experimental results are shown in Tables 5 and 6. Notably, the LiDAR system in the Ford AV Datasets¹⁹ consists of four Velodyne 32E sensors. To facilitate our experiments, we transformed their coordinate systems and fused the four 32-line LiDARs into an equivalent 128-line LiDAR. Based on this fused point cloud, we performed point cloud-image fusion and conducted experiments using the CPC-CVPE method. Due to the nature of the Ford AV Datasets¹⁹, where most scenes consist of highways and other environments with similar features,it presents a more challenging benchmark for cross-view pose estimation. Therefore, the evaluation metrics are relatively lower. Our dataset demonstrates comparable accuracy to the KITTI¹⁸ benchmark, thereby validating its feasibility for ground-to-air cross-view pose estimation applications.

Table 5.

Pose estimation results on the KITTI¹⁸ Dataset.

Type of ground data	Method	↓Location(m)		↑Lateral(%)		↑Longitudinal (%)		↓Heading(°)
		mean	median	r@1m	r@5m	r@1m	r@5m	mean	median
Colored point cloud	CPC-CVPE	1.03	0.55	97.87	99.79	86.21	97.98	1.01	0.65
Image	CCVPE12	1.22	0.62	97.35	99.71	77.13	97.16	0.67	0.54
	LDFF43	1.48	0.47	95.47	99.79	87.89	94.78	0.49	0.30
	SliceMatch44	7.96	4.39	49.09	98.52	15.19	57.35	4.12	3.65
	LM45	12.08	11.42	35.54	80.36	5.22	26.13	3.72	2.83
	DSM46	—	—	10.12	48.24	4.08	20.14	—	—
Point cloud	RSL-Net33	3.71	—	—	—	—	—	1.67	—
	Tang34	4.37	—	—	—	—	—	1.59	—
	Tang35	3.76	—	—	—	—	—	3.36	—

Bold numbers present the bset score.

Table 6.

Pose estimation results on the Ford AV Dataset¹⁹.

Area	Method	↓Location(m)		↑Lateral(%)		↑Longitudinal(%)		↓Heading (°)
		mean	median	r@1m	r@5m	r@1m	r@5m	mean	median
Log1	CPC-CVPE	9.67	9.78	59.31	95.12	6.77	26.12	2.47	2.48
	LDFF43	10.55	10.19	68.67	95.48	5.48	25.86	1.43	0.47
	CCVPE12	11.34	10.21	51.19	92.36	6.69	30.00	1.28	0.93
	LM45	12.54	12.63	48.57	71.57	5.90	26.33	3.13	1.29
	DSM46	—	—	12.00	53.67	4.33	21.43	—	—

Bold numbers present the bset score.

Discussion of limitation

Due to the roof-mounted configuration of the LiDAR sensor on the field robot, the collected point clouds lack detailed structural information from building facades above 5 meters in height. This limitation arises from the fixed vertical field-of-view (FoV) of the LiDAR 25^° downward tilt, which prioritizes ground-level feature capture for navigation tasks but omits upper building elements (e.g., windows, signage). The current dataset is exclusively collected under clear weather conditions. Adverse environments (e.g., rain, fog, snow) were not included due to the LiDAR’s susceptibility to precipitation-induced noise and the camera’s reduced visibility in low-light scenarios.

The platform’s GNSS configuration warrants specific clarification: The exclusion of RTK positioning stems from its empirically documented unreliability in environments characterized by urban canyons (building height-to-street width ratio  > 0.5) and dense arboreal coverage (canopy closure >70%)^38–41, where multipath signal reflections from building facades and vegetation-induced GNSS signal attenuation respectively induce decimeter-level ranging errors. In our experimental setup, the integration of RTK was constrained by the existing hardware architecture of the field robot system, which would necessitate additional specialized equipment beyond current configuration. However, we introduced the calculation method of the ground truth in Method, and proved that the method also achieves centimeter-level accuracy, which meets the accuracy requirements of this cross-view pose estimation task.

Usage Notes

The proposed McPed²³ is offered for use by academics for multi-modality cross-view pose estimation to facilitate the development of field robots navigation and even autonomous driving. This is the first multi-modality cross-view pose estimation dataset that can be used as a benchmark to facilitate the development of new algorithms. Because this dataset is aimed at cross-view pose estimation, we do not provide semantically related labels such as segmentation and detection. Researchers can annotate images or points accordingly for supervised learning.

Author contributions

X.Y., K.W. and J.X. designed the research. K.W. and R.Q. collected data. K.W. and J.X. did verification and calibration. J.X. solved the problem of data synchronization. K.W. and R.Q did annotation. R.Q. conducted the CPC-CVPE experiments and analyzed the results. All authors reviewed the manuscript.

Code availability

We provide a development kit programmed with Python and C++ language for this dataset, which contains scripts for visualizing and parsing the dataset. The toolkit is available at the code repository⁴². We provide some example files containing images, point clouds, npy files, internal and external parameter calibration files. The color_point.py provides the function of projecting the image RGB information into the point cloud through the camera internal reference and the external reference between LiDAR and camera to get the visualized color point cloud and get the corresponding npy preview results. The save_npy.py is used to generate and save the npy file,elevation and density maps can be generated separately.The path_visualization.cpp and path_visualization.h provide a path visualization method to check the accuracy of annotations.

Competing interests

The authors declare no competing interests.