Highly accurate data is crucial for the development and validation of automated driving. These are, for example, scenarios measured in real traffic or reference measurements for the environment perception of a prototype. fka has developed approaches to provide highly accurate measurement data using unmanned aerial vehicles. With this data, it is possible to generate scenarios and to provide a vehicle-independent reference measurement for environment perception.
The development and validation of highly automated driving functions is increasingly based on data-driven methods. Large data sets in the form of trajectories of road users measured in real traffic are crucial for various application areas in the development process of automated driving. These are, for example, models for behavior prediction of road users or scenario-based evaluation in the safety validation process. Such data sets are already urgently needed today and will become even more important in the future. In addition, the generation of highly accurate reference data to prove that the sensor technology of the highly automated vehicle captures the environment with sufficient accuracy and reliability is an unsolved challenge. To address both challenges, fka has developed a new method under the name levelXdata [1] to generate such data from Unmanned Aerial Vehicles (UAV, drones) equipped with high-resolution cameras. The data has a wide range of applications in the field of automated driving as reference data and trajectory data, Figure 1.
Figure 1.
Applications of drone data for automated driving, based on model A [2] (© fka)
To generate the data, the drone records high-resolution videos of public traffic events or a testing scenario from a bird's eye view. With the help of semantic segmentation and tracking algorithms, the positions and movements of all visible road users can be extracted with high precision from the video recordings. The drone's perspective allows the complete traffic scene to be captured without vehicles being occluded by other vehicles. In addition, measurements using a drone are very efficient due to the simultaneous capturing of all road users within the field of view. Figure 2 shows a comparison of the advantages and disadvantages of the approach with other measurement methods.
Figure 2.
Comparison of current traffic measurement methods (© fka)
One problem that arises when using infrastructure sensor technology or prototype vehicles with automation function with visible reference sensor technology (for example roof boxes) is the possible influence on the behavior of surrounding road users. A drone with an appropriate flight altitude, on the other hand, is imperceptible from a vehicle and exerts no influence on the traffic. It is thus a unique advantage of the drone measurement methodology that road users do not deviate from their natural behavior.
Application for Reference Measurement
Due to the combined use of different sensor types and processing algorithms, the environment perception of automated vehicles is an elementary but very complex module. In order to parameterize and evaluate this module, it is necessary to compare both the individual components and the entire perception system with a reference. Not only the detection rate and accuracy, but also the quality of the object tracking or prediction must be evaluated.
Precise ground truth data from a reference measurement system is required for the evaluation of the environment perception. Since the sensor data depends very much on the sensors used and their position on the vehicle, the evaluation must be carried out individually for each system. Thus, a test vehicle with the sensor setup to be investigated (Vehicle Under Test, VUT) must be used and further reference sensor data must be recorded in addition to the sensor data of the VUT. Assuming that the static environment is available in high-resolution digital maps, the following requirements for a reference measurement system can be derived for the dynamic environment on test tracks or in real traffic:
The measurement method must detect all relevant road users in the field of view of the VUT's sensor setup.
The accuracy of the reference measurement data should be consistently better than that of the system under test.
For use on a larger test track or in real traffic, the reference measurement system must cover a large measurement range or move with the VUT.
The cost should be reasonable and ideally should not scale with the area covered or the number of road users observed.
With respect to these requirements on reference data, the new drone measurement methodology is compared with the use of systems for position and motion detection (IMU-RTK GNSS systems) and additional on-board sensor technology such as high-resolution 360°-multilayer laser scanners, Figure 3.
Figure 3.
Comparison of reference measurement methods (© fka)
The use of IMU-RTK GNSS systems enables the unobtrusive recording of highly precise position data independent of a fixed location. However, the cost of precise IMU-RTK GNSS equipment is very high and must be multiplied by the number of road users to be measured simultaneously. Finally, uninvolved road users cannot be detected and the attachment of IMU-RTK GNSS devices to pedestrians or cyclists is difficult. The use of IMU-RTK GNSS technology is thus particularly advantageous for test scenarios with a maximum of one to two object vehicles.
The use of additional vehicle sensor technology, for example as a roof box, allows the environment to be captured from the same or a similar perspective as the sensors under test. The additional sensor technology and offline processing can generate accurate data, but occlusions still exist. In addition, high-resolution sensors are expensive and conspicuous, influencing surrounding traffic. Test vehicles for automation level 4 often already use an extensive and expensive 360°-perception for the automated driving system. The accuracy of those systems can only be surpassed by further sensor technology from a similar perspective at a high cost. The use is thus particularly advantageous for test vehicles up to automation level 3 in scenarios with little occlusion, if a high variety of different measurement locations is also required.
The use of a drone enables the detection of all road users with a high accuracy over the complete measurement range. Traffic is not affected by the measurement and the use of a drone is more cost-effective than the use of on-board 360°-laser sensors (lidar). However, a stationary flying drone can only cover a limited area in this case, which means that the test vehicle repeatedly moves out of the measurement area. This may be sufficient for particularly relevant locations such as intersections or roundabouts. However, to provide a permanent reference, the drone must actively follow the test vehicle, Figure 4.
Figure 4.
Reference measurement with a drone (© fka)
The fka has developed a drone-based reference measurement system for application on test tracks and in real traffic, which corresponds to the concept described before. In order for the drone to follow a test vehicle, it needs the exact position of the test vehicle. This information can either be provided via Global Navigation Satellite System (GNSS) positioning or extracted by real-time image processing of the drone camera image. Based on this, the drone adjusts its own position and orientation to that of the test vehicle. The analysis of this system on a test track with multiple vehicles equipped with an IMU-RTK GNSS system has shown that the road user position errors for both the drone and the IMU-RTK GNSS technology are in the single-digit centimeter range. The cover image of this article shows an example of reference measurements using a tracking flight in real traffic.
Trajectory and Scenario Data Sets for Development
In real traffic, a stationary flying drone can be used for traffic data collection. For a scenario-based safety validation process, traffic data sets are required in order to be able to extract the relevant and critical traffic scenarios in a data-driven manner. This was done, for example, in the Japanese Sakura project [3] and the German Pegasus project [4].
The new UNECE Regulation R157 "Automated Lane-Keeping Systems," which represents the first internationally agreed framework for the introduction of passenger cars with SAE level-3 systems, also requires scenario-based testing of the systems to be introduced and provides a scenario catalog. Since no concrete parameters are specified for this scenario catalog, fka compared selected scenarios of the regulation to recorded measurement data and extracted scenario parameters [5]. For this purpose, the dataset of German highways (highD, highway drone dataset) [6], which was created in 2018 with the drone measurement methodology, was used.
To consider different challenges and operational design domains, after the highD dataset, fka created some more datasets from different countries in North America, Asia or Europe and from various challenging locations like highway ramps, roundabouts and urban intersections. The new so-called uniD dataset [7], Figure 5, was created from measurements in Aachen at a public street in front of the main building of the RWTH Aachen University. Due to the fact that at this location many students are traveling alone or in groups and on foot or with a bicycle, many relevant interaction scenarios with vehicles arise. Measurements took place in 2019 prior to the Covid-19 pandemic outbreak.
Figure 5.
uniD dataset: interactions between vehicles and non-motorized road users (© fka)
To support research, the uniD dataset will again be made available free of charge for non-commercial public research. In recent years, such research support by levelXdata has already been provided for the highD dataset [6], the intersection dataset inD (intersection drone dataset) [8], and the roundabout dataset rounD (roundabout drone dataset) [9]. A license to use these datasets can be requested online at www.drone-dataset.com.
Summary and Outlook
Current methods for creating trajectory data sets and generating reference data for environment perception of automated vehicles are not sufficient for complex scenarios. Using a drone for measurement offers several advantages compared to existing approaches. Accordingly, drones are an effective tool for generating datasets for both highways and urban areas. Real-world traffic data sets include high-precision trajectories of road users, which can be used for data-driven development approaches as well as for safety validation of automated driving functions. For the evaluation of sensor systems, drones are particularly suitable for complex scenarios with more than two objects. By the automatic processing of the drone video recordings, measurement data with the quality of an RTK GPS (Real Time Kinematic using Global Positioning System) can be generated for a significantly higher number of road users at lower costs.
References
fka (eds.): levelXdata - Highly Accurate Traffic Data Captured from Aerial Perspective for all Levels of Automated Driving. Online: https://www.levelxdata.com/, access: February 23, 2021
Lampe, B.; Woopen, T.; Eckstein, L.: Collective Driving - Cloud Services for Automated Vehicles in Unicaragil. 28th Aachen Colloquium Automotive and Engine Technology, Aachen, 2019
Antona-Makoshi, J.; Uchida, N.; Yamazaki, K.; Ozawa, K.; Kitahara, E.; Taniguchi, S.: Development of a safety assurance process for autonomous vehicles in Japan. ESV Conference, Eindhoven, 2019
Winner, H.; Lemmer, K.; Form, T.; Mazzega, J.: Pegasus - first steps for the safe introduction of automated driving. In: Road Vehicle Automation 2019, No. 5
Tenbrock, A.; et al.: The ConScenD Dataset: Concrete Scenarios from the highD Dataset According to ALKS Regulation UNECE R157 in OpenX. Institute of Electrical and Electronics Engineers (IEEE) IV, 2021 (submitted)
Krajewski, R.; Bock, J.; Kloeker, L.; Eckstein, L.: The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems. 21th IEEE International Conference on Intelligent Transportation Systems (IEEE ITSC), Maui, 2018
Moers, T.; Vater, L.; Krajewski, R.; Bock, J.: The uniD Dataset: A Real-World Trajectory Dataset of Highly Interactive Scenarios in Germany. 24th IEEE International Conference on Intelligent Transportation Systems (IEEE ITSC), 2021 (submitted)
Krajewski, R.; Bock, J.; Moers, T.; Runde, S.: The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections}. IEEE International Conference on Connected Vehicles & Expo (IEEE ICCVE), 2019
Krajewski, R.; Bock, J.; Moers, T.; Vater, L.: The rounD Dataset: A Drone Dataset of Road User Trajectories at Roundabouts in Germany. 23rd IEEE International Conference on Intelligent Transportation Systems (IEEE ITSC), Rhodes, 2020
Biographies
Julian Bock, M. Sc.
is Manager Artificial Intelligence within the Business Unit Automated Driving at fka GmbH in Aachen (Germany).
Lennart Vater, M. Sc.
is Scientific Employee within the Research Area Vehicle Intelligence & Automated Driving at the Institute for Automotive Engineering (ika) of RWTH Aachen University (Germany).
Robert Krajewski, M. Sc.
is Specialist Machine Learning within the Research Area Vehicle Intelligence & Automated Driving at the Institute for Automotive Engineering (ika) of RWTH Aachen University (Germany).
Tobias Moers, M. Sc.
is Specialist Artificial Intelligence within the Business Unit Automated Driving at fka GmbH in Aachen (Germany).