Accelerated Evaluation of Automated Vehicles Safety in Lane-Change Scenarios Based on Importance Sampling Techniques

Abstract

Automated vehicles (AVs) must be thoroughly evaluated before their release and deployment. A widely used evaluation approach is the Naturalistic-Field Operational Test (N-FOT), which tests prototype vehicles directly on the public roads. Due to the low exposure to safety-critical scenarios, N-FOTs are time consuming and expensive to conduct. In this paper, we propose an accelerated evaluation approach for AVs. The results can be used to generate motions of the other primary vehicles to accelerate the verification of AVs in simulations and controlled experiments. Frontal collision due to unsafe cut-ins is the target crash type of this paper. Human-controlled vehicles making unsafe lane changes are modeled as the primary disturbance to AVs based on data collected by the University of Michigan Safety Pilot Model Deployment Program. The cut-in scenarios are generated based on skewed statistics of collected human driver behaviors, which generate risky testing scenarios while preserving the statistical information so that the safety benefits of AVs in nonaccelerated cases can be accurately estimated. The cross-entropy method is used to recursively search for the optimal skewing parameters. The frequencies of the occurrences of conflicts, crashes, and injuries are estimated for a modeled AV, and the achieved accelerated rate is around 2000 to 20 000. In other words, in the accelerated simulations, driving for 1000 miles will expose the AV with challenging scenarios that will take about 2 to 20 million miles of real-world driving to encounter. This technique thus has the potential to greatly reduce the development and validation time for AVs.

I. Introduction

Automated Vehicle (AV) technologies are actively studied by many companies because of their potential to save fuel, reduce crashes, ease traffic congestion, and provide better mobility, especially to those who cannot drive [1]. Currently, almost all major automakers have research and development programs on AVs. By 2030, it is estimated that the sales of AVs may reach $87 billion dollars [2].

National Highway Traffic Safety Administration defines five levels of AV automation [3]. AVs are quickly being developed from level 0 automation, which conducts no driving tasks and up, possibly all the way to level 4 automation, which monitors the driving environment performs all dynamic driving duties. As the level of automation goes up, AVs need to deal with many uncertainties/disturbances in the real world, including imperfect human driver behaviors. AVs are projected to penetrate the market gradually and will co-exist with human-controlled vehicles (HVs) for decades [4]. During this transitional period, AVs will interact primarily with HVs. It is estimated that 70–90% of motor vehicle crashes are due to human errors [5], [6]. However, AVs can have their own crash modes. A practical and effective evaluation of the safety performance of AVs should consider their interactions with HVs.

Approaches for AV evaluation can be summarized into four categories as shown in Fig. 1. One approach to studying the interactions between AVs and HVs is through Naturalistic Field Operational Tests (N-FOT) [7]. In an N-FOT, data is collected from a number of equipped vehicles driven under naturalistic conditions over an extended period of time [8]. Several N-FOT projects [9]–[16] have been conducted in the U.S. and Europe. Conducting an N-FOT to evaluate an AV function typically involves non-intrusive conditions, i.e., the test drivers were told to drive as they normally do on public roads. This test approach suffers from several limitations. An obvious problem is the time needed. Under naturalistic conditions, the level of exposure to dangerous events is very low. In the U.S., there were 5.7 million police-reported motor vehicle crashes and 30 057 fatal crashes in 2013, while the vehicles traveled a total of 2.99 trillion miles [17]. This translates to approximately 0.53 million miles for a police-reported crash and 99 million miles for a fatal crash. Since the average mileage driven annually by licensed drivers is 14 012 miles [17], one needs to drive on average 38 years to experience a police-reported crash and 6877 years for a fatal crash. Because of this low exposure rate, the N-FOT projects need a large number of vehicles, long test duration, and a large budget. According to Akamatsu et al. [18], an N-FOT “cannot be conducted with less than $10 000 000.” A more efficient approach for AV evaluation is needed.

Fig. 1.

Summary of evaluation approaches for AVs.

Some researchers built stochastic models based on the big data obtained from N-FOTs and ran Monte Carlo simulations to evaluate AVs. Yang et al. [19] and Lee [20] evaluated collision avoidance systems by replaying segments extracted from the Road-Departure Crash-Warning (RDCW) FOT and Intelligent Cruise Control (ICC) FOT naturalistic driving databases. Woodrooffe et al. [21] generated 1.5 million forward collision scenarios based on naturalistic driving conflicts and used them to evaluate collision warning and collision mitigation braking technologies on heavy trucks. Reusing the N-FOT data in simulations can avoid the large budget for N-FOTs. However, even for computer simulations, low exposure to safety critical scenarios is still an issue.

The test matrix approach has been the basis of many test procedures, such as the AEB (Autonomous Emergency Braking) test protocol [22] of the Euro New Car Assessment Program (Euro-NCAP). Much development work was done to advance this evaluation approach including CAMP [23], HASTE [24], AIDE [25], TRACE [26], APROSYS [27] and ASSESS [28]. The test scenarios are frequently selected based on national crash databases [29], such as GES (General Estimates System) [30], NMVCCS (National Motor Vehicle Crash Causation Survey) [31] and EDR (Event Data Recorder databases) [32]. A systematic review of this approach can be found in [8]. The main benefits of this test method are that it is repeatable, reliable, and can be finished in a reasonable amount of time. However, it is not clear how the selected test scenarios correlate with real-world conditions, especially when human interaction is involved [8], [33]. Moreover, because all test scenarios are fixed and predefined, AVs can be tuned to achieve good performance in these tests, but their behaviors under broader conditions are not adequately assessed [34].

Another approach, the Worst-Case Scenario Evaluation (WCSE) methodology, has been studied by Ma et al. [35], Ungoren et al. [36] and Kou [37] to identify the most challenging scenarios using model-based optimization techniques. While the worst-case evaluation method can identify the weakness of a vehicle control system, it does not consider the probability of occurrence of the worst-case scenarios. There- fore, the worst case evaluation results do not provide sufficient information about the risk in the real world and may not be the fairest way to compare different designs.

In a previous work [38], we proposed the accelerated evaluation concept and applied it to the car-following scenarios. The crash rate in the real world was estimated based on the national crash database. In [39], we introduced the Importance Sampling techniques to improve the reliability and accuracy of the estimation, in which the parameters in the accelerated tests were tuned by hands. In this paper, we further proposed an automated method to search for the best way to morph the original lane change behavior statistics. As shown in Fig. 2, first, HVs are modeled based on data extracted from N-FOT databases to represent the human driving behaviors. Second, an accelerated model is constructed by modifying the probability density functions of the stochastic variables to promote riskier lane change behaviors. Third, the optimal parameters of the accelerated model are obtained through an iterative search. Finally, the “amplified” results together with the statistics in the accelerated model are used to calculate the performance of the host vehicles in real world driving. The contribution of this paper is that we proposed the Accelerated Evaluation of AV procedure which provides high accuracy and accelerated evaluation using Importance Sampling theory and the Cross Entropy method. To the best of our knowledge, we are the first group to apply these techniques to create test scenarios to evaluate AV safety and calculate social benefits. The meaningfulness of doing this is not only to accelerate the simulation, but also to provide a way to objectively identify critical test scenarios that can be used in other types of evaluation platforms such as driving simulator, on-track tests, or hardware-in-the-loop tests.

Fig. 2.

Procedure of the accelerated evaluation method.

II. Lane Change Models Based on Naturalistic Driving

The lane change (cut-in) scenario is used as an example to show the benefits of the proposed accelerated evaluation approach. Lane change, defined as a vehicle moving from one lane to another in the same direction of travel [40], can cause a frontal collision crash for the following vehicle when the time gap is too short. Successful completion of a lane change requires attention to the vehicles in both the original lane and the adjacent lane [41]. In the US, there are between 240 000 and 610 000 reported lane-change crashes, resulting in 60 000 injuries annually [40]. Few protocols have been published regarding the evaluation of AVs (e.g., AEB systems) under lane change scenarios.

Human drivers’ lane change behaviors have been analyzed and modeled for more than half a century. Early studies based on controlled experiments usually have short test horizons and limited control settings [42]. More recently, researchers started to use large scale N-FOT databases to model the lane change behaviors. Lee et al. [42] examined steering, turn signal and brake pedal usage, eye glance patterns, and safety envelope of 500 lane changes. The 100-Car Naturalistic Driving Study analyzed lane change events leading to rear-end crashes and near-crashes [40]. Zhao et al. [43] analyzed the safety critical variables in mandatory and discretionary lane changes for heavy trucks [12]. Most of these studies are based on hundreds of lane changes. We use the data collected in the Safety Pilot Model Deployment (SPMD) project, which contains more than 400 000 lane changes.

A. Identification of the Lane Change Events

In this research, we developed a lane change statistical model and demonstrated its use for accelerated evaluation of a frontal collision avoidance algorithm. The data used is from the Safety Pilot Model Deployment database [44]. The SPMD program aims to demonstrate connected vehicle technologies in a real-world environment. It recorded naturalistic driving of 2842 equipped vehicles in Ann Arbor, Michigan for more than two years. As of April 2015, 34.9 million miles were logged, making SPMD one of the largest public N-FOT databases ever.

As shown in Fig. 3, a lane change was detected and recorded by an SPMD vehicle when the Lane Change Vehicle (LCV) crosses the lane markers. In the SPMD program, 98 sedans are equipped with Data Acquisition System and MobilEye [45], which provides: a) relative position to the lane change vehicle (range), and b) lane tracking measures pertaining to the lane delineation both from the painted boundary lines and road edge characteristics. The error of range measurement is around 10% at 90 m and 5% at 45 m [46].

Fig. 3.

Lane-change scenarios that may cause frontal crashes.

To ensure consistency of the used dataset, the following criteria were applied:

$$•\upsilon (t_{\mathrm{L}\mathrm{C}})\in (2\mathrm{m}/\mathrm{s},40\mathrm{m}/\mathrm{s})$$

$$•{\upsilon }_L(t_{\mathrm{L}\mathrm{C}})\in (2\mathrm{m}/\mathrm{s},40\mathrm{m}/\mathrm{s})$$

$$•R_L(t_{\mathrm{L}\mathrm{C}})\in (0.1\mathrm{m},75\mathrm{m})$$ (1)

where t_LC is the time when the center line of the LCV crosses the lane markers; υ_L and υ are the velocities of the LCV and the SPMD vehicle; R_L is the range defined as the distance between the rear edge of the LCV and the front edge of the SPMD vehicle. 403 581 lane changes were detected in total. Fig. 4 shows the locations of the identified lane changes.

Fig. 4.

Recorded lane-change events from the SPMD database.

B. Lane Change Models

A lane change can be divided into three phases: the decision to initiate a lane change, gap (range) acceptance, and lane change execution [42]. In this research, we focus on the effects of gap acceptance, which is mainly captured by three variables: υ_L(t_LC), R_L(t_LC) and Time To Collision (TTC) of AVs, defined as

$$\mathrm{T}\mathrm{T}{\mathrm{C}}_L=-\frac{R_L}{{Ṙ}_L}$$ (2)

where Ṙ_L is the derivative of R_L. In the following, unless mentioned specifically, υ_L, R_L and TTC_L are the variables at t_LC.

The distribution of υ_L is shown in Fig. 5. The division of highways and local roads is embodied by the bimodal shape of the histogram. υ_L is assumed to remain constant during the lane change. Only the events with a negative range rate are used to build the lane change model. Out of 403 581 lane change events, 173 692 are with negative range rate.

Fig. 5.

Distributions of υ_L(t_LC) of lane-change events used in our model.

Larger R_L and TTC_L indicate the scenario is safer which are the majority cases in naturalistic driving, while Smaller R_L and TTC_L indicate the scenario is less safe and rarer. Therefore, we define the variables of interest as reciprocal of R_L and TTC_L to put the rare events in the tail of the distribution to naturally fit the naturalistic driving statistics. To capture the influence of vehicle speed on range and TTC, we divided lane change events into low, medium. and high velocity conditions. Fig. 6 shows that υ_L has little influence on the distribution of $R_L^{-1}$. We use a standard Matlab package [47] to search for a proper distribution to fit $R_L^{-1}$, which examines 17 different types of distributions and examine goodness-of-fit by using Bayesian Information Criterion [48]. Fig. 7 illustrates the fitting of $R_L^{-1}$ using a Pareto distribution defined as

$$f_{R_L^{-1}}\left(x\right|k_{R_L^{-1}},{\sigma }_{R_L^{-1}},{\theta }_{R_L^{-1}})=\frac{1}{{\sigma }_{R_L^{-1}}}{(1+k_{R_L^{-1}}\frac{x-{\theta }_{R_L^{-1}}}{s_{R_L^{-1}}})}^{-1-1/k_{R_L^{-1}}}$$ (3)

where the shape parameter $k_{R_L^{-1}}$, the scale parameter ${\sigma }_{R_L^{-1}}$, and the threshold parameter ${\theta }_{R_L^{-1}}$ are all positive. Note that, due to the physical limitations mentioned in (1), the Pareto distribution in (3) is in fact truncated at 1/0.1 m⁻¹ and 1/75 m⁻¹. For the sake of conciseness, we show the untruncated version throughout this paper. The same holds for all other fitted distributions in this paper.

Fig. 6.

Distributions of $R_L^{-1}(t_{\mathrm{L}\mathrm{C}})$ at different vehicle forward speeds.

Fig. 7.

Fitting results of $R_L^{-1}(t_{\mathrm{L}\mathrm{C}})$ using the Pareto distribution.

The histograms of $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ for different velocity intervals are shown in Fig. 8. As the vehicle speed increases, the mean of $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ decreases. Based on the analysis using MATLAB fitting package [47], $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ can be approximated by both Pareto distribution and exponential distribution with 0.23% relative difference in BIC. We used the exponential distribution

$$f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}\left(x\right|{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}})=\frac{1}{{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}{e}^{-x/{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}$$ (4)

for simplicity, where the scaling factor ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ varies with the speed of the LCV. Here we define ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ as the mean value instead of the rate of the exponential distribution, because mean value has more intuitive physical meaning.

Fig. 8.

Distribution of $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}(t_{\mathrm{L}\mathrm{C}})$ at different lane-change vehicle speeds.

The dependence of ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ on vehicle speed is shown in Fig. 9. As the vehicle speed increases, ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ decreases. The blue circles represent ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ at the center points of υ_L intervals. We use linear interpolation and extrapolation to create smooth ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ for all vehicle speeds.

Fig. 9.

Model parameters for TTC_L(t_LC).

The effect of range on TTC is very limited, as can be seen in Fig. 10. This indicates that R_L and TTC_L can be modeled independently given the same υ_L. Ṙ_L can then be calculated from (5)

$${Ṙ}_L=-\frac{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}{R_L^{-1}}.$$ (5)

Finally, the velocity of the host vehicle υ can be calculated from

$$\upsilon ={\upsilon }_L-{Ṙ}_L.$$ (6)

Fig. 10.

Distribution of TTC_L(t_LC) at different range intervals.

In summary, the lane change events are generated in the following order: a) generate υ_L based on the empirical distributions shown in Fig. 5; b) generate $R_L^{-1}$ using Fig. 7; c) generate $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ using the Exponential distribution with parameters shown in Fig. 9; and finally d) calculate υ using (5) and (6).

III. Accelerated Evaluation

Monte Carlo techniques can be used to simulate driving conditions using a stochastic model, but a naïve implementation will take a long time to execute. The key of accelerated evaluation is to skew the statistics of the Monte Carlo samples but still be able to maintain statistical correctness and accuracy. In this section, we first show the limitation of the “crude” Monte Carlo (CMC) in simulating events with small probability (rare events). We then introduce the Importance Sampling (IS) concept. Thirdly, we show how to apply IS to evaluate AVs in lane change scenarios. Finally, we introduce the Cross Entropy method to optimize the use of IS.

A. Monte Carlo Estimation

Monte Carlo method [49] typically aims to generate unbiased statistical samples to estimate the expected value of a stochastic process. Let Ω be the sample space for all possible events, and ℇ ⊂ Ω be the rare events of interest, e.g., the occurrence of a crash. Let x be a random vector describing the motions of the lane change vehicle. The indicator function of the event ℇ is defined as

$$I_{ℰ}(\mathit{x})=\{\begin{array}{cc}1,\hfill & \text{if }\mathit{x}\in ℰ\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$ (7)

Our goal is to estimate the probability of ℇ happening, i.e.,

$$\gamma =P(ℰ)=E(I_{ℰ}(\mathit{x})).$$ (8)

The CMC approach generates independent and identically distributed samples x₁, x₂, …, x_n of x, and then calculate the sample average

$${\hat{\gamma }}_n=\frac{1}{n}\sum _{i=0}^n{I}_{ℰ}({\mathit{x}}_i).$$ (9)

We state some statistical properties of CMC. First, under mild conditions, the Strong Law of Large Numbers [49] holds, i.e.,

$$P(\underset{n\to 8}{\text{lim}}{\hat{\gamma }}_n=\gamma )=1.$$ (10)

Moreover, the Central Limit Theorem [49] implies that, when n is large, γ̂_n follows approximately the normal distribution 𝒩(E(γ̂_n), σ²(γ̂_n)) with the mean

$$E({\hat{\gamma }}_n)=E(\frac{1}{n}\sum _{i=0}^n{I}_{ℰ}({\mathit{x}}_i))=\gamma$$ (11)

and variance

$${\sigma }^2({\hat{\gamma }}_n)=\text{Var}({\hat{\gamma }}_n)=\text{Var}(\frac{1}{n}\sum _{i=0}^n{I}_{ℰ}({\mathit{x}}_i))=\frac{1}{n^2}\sum _{i=0}^n\text{Var}(I_{ℰ}(x_i))=\frac{\gamma (1-\gamma )}{n}.$$ (12)

The accuracy of the estimation is represented by the relative half-width, which is the half-width of the confidence interval relative to the probability to be estimated. With the Confidence Level at 100 (1 − α)%, the relative half-width of γ̂_n is defined as

$$l_r=\frac{l_{\alpha }}{\gamma }$$ (13)

where l_α is the half-width given by

$$l_{\alpha }=z_{\alpha }\sigma ({\hat{\gamma }}_n)$$ (14)

and z_α is defined as

$$z_a={\mathrm{\Phi }}^{-1}(1-\alpha /2)$$ (15)

where Φ⁻¹ is the inverse cumulative distribution function of 𝒩(0,1). To ensure l_r is smaller than a constant β, we need

$$\frac{l_{\alpha }}{\gamma }=\frac{z_{\alpha }\sigma ({\hat{\gamma }}_n)}{\gamma }=\frac{z_{\alpha }}{\gamma }\sqrt{\frac{\gamma (1-\gamma )}{n}}=z_a\sqrt{\frac{1-\gamma }{\gamma n}}\le \beta .$$ (16)

In other words

$$n\ge \frac{z_{\alpha }^2}{{\beta }^2}·\frac{1-\gamma }{\gamma }.$$ (17)

Equation (17) reveals that when ℇ is rare, i.e., γ0, the required test number n goes to infinity. This means that a huge test number is required to maintain a satisfactory half-width relative to the magnitude of a rare event probability γ. This is the reason why CMC is slow.

B. Importance Sampling (IS)

IS is a so-called variance reduction technique that is effective in handling rare events. IS has been successfully applied to evaluate critical events in reliability [50], finance [51], insurance [52], and telecommunication networks [53]. General overviews about IS can be found in [54]–[56].

To explain the concept of IS, we denote f(x) as the original joint density function of the random vector x. The core idea of IS is to replace f(x) with a new density f*(x) that has a higher likelihood for the rare events to happen. Using a different distribution, however, leads to biased samples, and the key of IS is to provide a mechanism to compensate for this bias and compute correct crash rate at the end.

We describe this mechanism as follows. First, we define the likelihood ratio L (Radon-Nikodym derivative [57]) as

$$L(x)=\frac{f(\mathit{x})}{f^{*}(\mathit{x})}.$$ (18)

The probability of E satisfies

$$P(ℰ)=E_f(I_{ℰ}(\mathit{x}))=\int I_{ℰ}(\mathit{x})f(\mathit{x})d\mathit{x}=\int [I_{ℰ}(\mathit{x})L(\mathit{x})]f^{*}(\mathit{x})\mathrm{d}\mathit{x}=E_{f*}(I_{ℰ}(\mathit{x})L(\mathit{x})).$$ (19)

One required condition for (19) to hold is that f*(x) must be absolutely continuous with respect to f(x) within ℇ, i.e.,

$$\forall x\in ℰ:f^{*}(\mathit{x})=0\Rightarrow f(\mathit{x})=0$$ (20)

which guarantees the validity of L in (18). The IS sample is I_ℇ(x_i)L(x_i) where x_i is generated under f*(x), which is an unbiased estimator for γ. The overall IS estimator for test number n is then

$${\hat{\gamma }}_n=\frac{1}{n}\sum _{i=0}^n{I}_{ℰ}({\mathit{x}}_i)L({\mathit{x}}_i).$$ (21)

Note that although a continuous model is used in this paper, similar approaches can be applied to the discrete model as well.

Now consider the relative half-width of CI constructed by IS

$$\begin{gathered} l_r^{*}=\frac{l_{\alpha }}{\gamma }=\frac{(z_{\alpha }\sigma s({\hat{\gamma }}_n)}{\gamma }=\frac{z_a\sqrt{(E_{f*}({\hat{\gamma }}_n^2)-E_{f*}^2({\hat{\gamma }}_n)}}{\gamma \sqrt{n}} \\ =\frac{z_{\alpha }\sqrt{E_{f*}(I_{ℰ}^2(x)L^2(x))-{\gamma }^2}}{\gamma \sqrt{n}} \\ =\frac{z_{\alpha }}{\sqrt{n}}\sqrt{\frac{E_{f*}(I_{ℰ}^2(\mathit{x})L^2(\mathit{x}))}{{\gamma }^2}-1}\le \beta . \end{gathered}$$ (22)

The required minimum test number is then

$$n\ge \frac{z_{\alpha }^2}{{\beta }^2}(\frac{E_{f*}(I_{ℰ}^2(\mathit{x})L^2(\mathit{x}))}{{\gamma }^2}-1).$$ (23)

When f*(x) is properly chosen, $E_{f*}(I_{ℰ}^2(\mathit{x})L^2(\mathit{x}))$ can be close to γ², resulting in a smaller number of tests (i.e., the evaluation is accelerated).

C. Accelerated Evaluation of Automated Vehicles in Lane Change Scenarios

When a slower lane changing vehicle cut in front of the AV, the events of interest are defined as

$$ℰ=\{\text{min}(R_L(t))<R_{ℰ}|t_{\mathrm{L}\mathrm{C}}<t\le t_{\mathrm{L}\mathrm{C}}+T_{\mathrm{L}\mathrm{C}}\}$$ (24)

where T_LC represents duration of the lane change event; R_ℇ is the critical range. Equation (24) means that if the minimum range is smaller than R_ℇ anytime during the lane change event, this lane change belongs to the ℇ set.

The random vector x consists of three random variables $[{\upsilon }_L,\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1},R_L^{-1}]$. υ_L is generated using the empirical distributions shown in Fig. 5. The IS approach considers the modified probability density functions of $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ and $R_L^{-1}$ denoted by $f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{*}(\mathit{x})$ and $f_{R_L^{-1}}^{*}(\mathit{x})$. The likelihood ratio is then

$$L(R_L^{-1}=x,\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}=y)=\frac{f_{R_L^{-1}}(x)f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(y)}{f_{R_L^{-1}}^{*}(x)f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{*}(y)}.$$ (25)

From (19), the probability of ℇ can be estimated as

$$P(ℰ)=E_f(I_{ℰ}(\mathit{x}))=E_{f*}(I_{ℰ}(\mathit{x})L(\mathit{x})).$$ (26)

The only task left is then to construct proper $f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{*}(\mathit{x})$ and $f_{R_L^{-1}}^{*}(\mathit{x})$ to accelerate the evaluation procedure.

D. Searching for Optimal IS Distributions With the Cross Entropy Approach

The choice of IS distribution is critical to the success of the IS method. The Cross Entropy (CE) method, first proposed by Rubinstein [58], is an iterative search procedure to find good IS distribution within a prescribed parametric family.

To understand how CE works, we first point out an important observation: the theoretical optimal IS distribution is always the conditional distribution given that the rare event of interest happens, namely

$$f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})=\{\begin{array}{cc}\frac{f(\mathit{x})}{\gamma },\hfill & I_{ℰ}(\mathit{x})=1\hfill \\ 0,\hfill & I_{ℰ}(\mathit{x})=0.\hfill \end{array}$$ (27)

With $f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})$, any sampled x leads to a rare event so that the indicator function I_ℇ (x) constantly equals to one. The likelihood ratio

$$L_{\mathrm{z}\mathrm{v}}(\mathit{x})=\frac{f(\mathit{x})}{f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})}=\gamma .$$ (28)

The probability of the rare events is calculated by

$${\hat{\gamma }}_n=\frac{1}{n}\sum _{i=0}^N{I}_{ℰ}({\mathit{x}}_n)L({\mathit{x}}_n)=\frac{1}{n}\sum _{i=0}^n\gamma =\gamma .$$ (29)

In other words, γ̂_n equals to γ for all n. The distribution $f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})$ is optimal in the sense that any sample generated from it has zero variance, and hence the required test number to construct confidence level to any precision is 1; thus it is also known as the zero variance IS distribution [56]. Unfortunately, this distribution cannot be implemented directly because it requires the knowledge of γ, which is exactly what we want to estimate. However, it provides a benchmark to get good IS distributions: A good IS distribution should be close to the zero-variance distribution.

To describe how CE works, we define the Kullback-Leibler (KL) divergence

$$f_{\mathrm{K}\mathrm{L}}({\tilde{f}}_{\vartheta }(\mathit{x}),f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x}))=\int \text{log }\left[\frac{f_z{\upsilon }^{*}(x)}{{\tilde{f}}_{\vartheta }(x)}\right]f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})d\mathit{x}$$ (30)

as a measure of the difference between f̃_ϑ (x) and $f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})$. The idea of CE is to find an IS distribution over the family of distributions f̃_ϑ(x) (controlled by ϑ) that has the minimum KL divergence with $f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})$, i.e.,

$${\mathit{\vartheta }}^{*}=\text{arg }\underset{\vartheta }{\text{min }}{f}_{\mathrm{K}\mathrm{L}}({\tilde{f}}_{\vartheta }(\mathit{x}),f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})).$$ (31)

Substituting (30) into (31), we have

$${\mathit{\vartheta }}^{*}=\text{arg }\underset{\vartheta }{\text{min}}\int \text{log }\left[\frac{f_{\mathrm{z}\mathrm{v}}^{*}(x))}{{\tilde{f}}_{\vartheta }(x)}\right]f_{\mathrm{z}\mathrm{v}}^{*}(x)dx=\text{arg }\underset{\vartheta }{\text{min}}\int \{\text{log}[f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})]f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})-\text{log}[{\tilde{f}}_{\vartheta }(\mathit{x})]f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})\}d\mathit{x}.$$ (32)

Note the first term inside the integration is independent of ϑ (32) can be simplified to

$${\mathit{\vartheta }}^{*}=\text{arg }\underset{\vartheta }{\text{max}}\int \text{log}[{\tilde{f}}_{\vartheta }(\mathit{x})]f_{\mathrm{z}\mathrm{v}}^{*}(\mathit{x})d\mathit{x}.$$ (33)

Substituting (27) into (33), we have

$${\mathit{\vartheta }}^{*}=\text{arg }\underset{\vartheta }{\text{max}}\int \text{log}[{\tilde{f}}_{\vartheta }(\mathit{x})]\frac{f(\mathit{x})}{P(ℰ)}{I}_{ℰ}(\mathit{x})d\mathit{x}=\text{arg }\underset{\vartheta }{\text{max}}\int \text{log }[{\tilde{f}}_{\vartheta }(\mathit{x})]f(\mathit{x})I_{ℰ}(\mathit{x})d\mathit{x}.$$ (34)

The CE method is an iterative scheme to sequentially improve the IS distribution and optimize ϑ* using (33). At the ith iteration, we use f̃_ϑi(x) as the IS distribution to run the Monte Carlo. Then, letting L̃_ϑi(x) = f(x)/ (f̃_ϑi (x), from (34), ϑ_i+1 can be derived as

$${\mathit{\vartheta }}_{i+1}=\text{arg }\underset{\vartheta }{\text{max}}\int \text{log}[{\tilde{f}}_{\vartheta }(\mathit{x})]{\tilde{L}}_{{\vartheta }_i}{I}_{ℰ}(\mathit{x}){\tilde{f}}_{{\vartheta }_i}(\mathit{x})d\mathit{x}\approx \text{arg }\underset{\vartheta }{\text{max}}{\mathrm{Ê}}_{{\tilde{f}}_{{\vartheta }_i}}\left[\text{log }\right({\tilde{f}}_{\vartheta }(\mathit{x})){\tilde{L}}_{{\vartheta }_i}(\mathit{x})I_{ℰ}(\mathit{x})]$$ (35)

where I_ℇ(x) are samples in the previous iteration and Ê_f̃ϑi [·] denotes the empirical average.

There are many possible choices for the family of f̃_ϑ(x). Here we use a popular class named the Exponential Change of Measure (ECM) for $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$.

Recall that $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}~\text{exp}({\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}({\upsilon }_L))$. ECM considers the family

$${\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x)=\text{exp}({\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}x-\mathrm{\Psi }({\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}))f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x)$$ (36)

parametrized by ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}$, where $\mathrm{\Psi }({\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}})$ is the logarithmic moment generation function of $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$, i.e.,

$$\Psi \left({\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}\right)=\text{log }\mathrm{E}\left(\text{exp}\right({\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}\left)\right).$$ (37)

It can be derived that

$${\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x)=(\frac{1}{{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}-{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}})\text{ exp }(-(\frac{1}{{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}-{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}\left)x\right)$$ (38)

where ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}<1/{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ and ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}>0$. To make ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}$ have the same scale as ${\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$, we apply a nonlinear mapping by letting

$${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^{\mathrm{E}\mathrm{C}\mathrm{M}}=\frac{{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}{{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}-{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}^2}$$ (39)

with ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}<{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$. Substitute (39) into (38), we have

$${\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}\left(x\right|{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}})=(\frac{1}{{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}-{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}\left)\text{ exp }\right(\frac{x}{{\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}-{\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}}).$$ (40)

$R_L^{-1}$ follows a (truncated) Pareto distribution, i.e.,

$$f_{R_L^{-1}}(x)=\text{Pareto }\left(x\right|k_{R_L^{-1}},{\sigma }_{R_L^{-1}},{\theta }_{R_L^{-1}}).$$ (41)

We apply an ECM of the exponential distribution as our family of IS distributions, where we first construct an exponential distribution

$${\tilde{f}}_{R_L^{-1}}(x)=\frac{1}{{\lambda }_{R_L^{-1}}}\text{exp }(-\frac{1}{{\lambda }_{R_L^{-1}}}x)$$ (42)

with ${\lambda }_{R_L^{-1}}$, which gives (42) the smallest least square error to (41). With similar procedure, we have

$${\tilde{f}}_{R_L^{-1}}(x|{\vartheta }_{R_L^{-1}})=(\frac{1}{{\lambda }_{R_L^{-1}}-{\vartheta }_{R_L^{-1}}}\left)\text{ exp }\right(-\frac{x}{({\lambda }_{R_L^{-1}}-{\vartheta }_{R_L^{-1}})}).$$ (43)

Using this approximate ECM instead of an ECM applied to a truncated Pareto reduces the computation complexity in the optimization step since a closed form can be obtained in each Cross Entropy iteration.

The overall likelihood ratio is

$$\tilde{L}(R_L^{-1}=x,\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}=y)=\frac{f(\mathit{x})}{{\tilde{f}}_{{\vartheta }_i}(\mathit{x})}=\frac{f_{R_L^{-1}}(x)f_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(y)}{{\tilde{f}}_{R_L^{-1}}(x){\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(y)}.$$ (44)

For low-velocity conditions, i.e.,

$${\upsilon }_L\in (5\mathrm{m}/\mathrm{s},15\mathrm{m}/\mathrm{s})$$

we simulate N tests with initial condition

$$\mathit{x}=[{\upsilon }_L,\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1},R_L^{-1}]$$

where υ_L follows the low velocity portion (5 m/s~15 m/s) of the empirical distribution shown in Fig. 5. $R_L^{-1}$ and $\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}$ follows ${\tilde{f}}_{R_L^{-1}}(x)$ and ${\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x)$. Apply (38) and (43) to (35). The optimal parameter ${\vartheta }_{R_L^{-1}}$ and ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ can be derived analytically

$${\vartheta }_{R_L^{-1}}=\frac{{\sum }_{j=1}^N\tilde{L}({\mathit{x}}_j)I_{ℰ}({\mathit{x}}_j)({\lambda }_{R_L^{-1}}-R_{L,j}^{-1})}{{\sum }_{j=1}^N\tilde{L}({\mathit{x}}_j)I_{ℰ}({\mathit{x}}_j)}$$ (45)

$${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}=\frac{{\sum }_{j=1}^N\tilde{L}({\mathit{x}}_j)I_{ℰ}({\mathit{x}}_j)({\lambda }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}-TT{C}_L^{-1})}{{\sum }_1^N\tilde{L}({\mathit{x}}_j)I_{ℰ}({\mathit{x}}_j)}$$ (46)

where j is the index for each simulation. The newly obtained ${\vartheta }_{R_L^{-1}}$ and ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ can be used in the next iteration.

The same procedure can be used to obtain optimal parameters in medium and high-velocity conditions.

IV. Simulation Analysis

An AV model was designed to demonstrate the proposed accelerated evaluation approach in the lane change scenarios.

A. Design of Test Automated Vehicle

The AV is designed to be equipped with both Adaptive Cruise Control (ACC) [59] and Autonomous Emergency Braking (AEB). When the driving is perceived to be safe (TTC_L ≥ TTC_AEB), it is controlled by the ACC. The ACC is approximated by a discrete Proportional-Integral (PI) controller [59] to achieve a desired time headway $T_{\mathrm{H}\mathrm{W}\mathrm{d}}^{\mathrm{A}\mathrm{C}\mathrm{C}}$. Use the time headway error $t_{\mathrm{H}\mathrm{W}}^{\text{Err}}$ as the controller input

$$t_{\mathrm{H}\mathrm{W}}^{\text{Err}}=t_{\mathrm{H}\mathrm{W}}-T_{\mathrm{H}{\mathrm{W}}_d}^{\mathrm{A}\mathrm{C}\mathrm{C}}$$ (47)

where t_HW is the current time headway, defined as

$$t_{\mathrm{H}\mathrm{W}}=\frac{R_L}{\upsilon }.$$ (48)

The discrete PI controller can be described in the discrete-time domain as

$$\frac{A_d(z)}{T_{\mathrm{H}\mathrm{W}}^{\text{Err}}(z)}=K_p^{\mathrm{A}\mathrm{C}\mathrm{C}}+K_i^{\mathrm{A}\mathrm{C}\mathrm{C}}\frac{T_s}{2}\frac{z+1}{z-1}$$ (49)

where A_d and $T_{\mathrm{H}\mathrm{W}}^{\text{Err}}$ are the Z transformation of the command acceleration a_d and $t_{\mathrm{H}\mathrm{W}}^{\text{Err}}$; T_s is the sampling time; gains $K_p^{\mathrm{A}\mathrm{C}\mathrm{C}}$ and $K_i^{\mathrm{A}\mathrm{C}\mathrm{C}}$ are calculated using the Matlab Control Toolbox using the following requirements: a) Loop bandwidth = 10 rad/s, and b) Phase margin = 60 degree. The control power of ACC system is saturated to a constant acceleration $a_{\mathrm{A}\mathrm{C}\mathrm{C}}^{\text{Max}}$, i.e., $|a_d|\le a_{\mathrm{A}\mathrm{C}\mathrm{C}}^{\text{Max}}$. To implement the PI controller in the time domain, taking the inverse Z transformation of (49), we get

$$a_d(k+1)=a_d(k)+K_p^{\mathrm{A}\mathrm{C}\mathrm{C}}(t_{\mathrm{H}\mathrm{W}}^{\text{Err}}(k)-t_{\mathrm{H}\mathrm{W}}^{\text{Err}}(k-1))+K_i^{\mathrm{A}\mathrm{C}\mathrm{C}}(t_{\mathrm{H}\mathrm{W}}^{\text{Err}}(k)+t_{\mathrm{H}\mathrm{W}}^{\text{Err}}(k-1))T_s/2.$$ (50)

The AEB model was extracted from a 2011 Volvo V60, based on a test conducted by ADAC (Allgemeiner Deutscher Automobil-Club e.V.) [60] (see Fig. 11). It is analyzed using test track data, owner’s manuals, European New Car Assessment Program (Euro NCAP) information, and videos during vehicle operation [61]. The AEB algorithm becomes active when TTC_L < TTC_AEB, where TTC_AEB depends on the vehicle speed as shown in Fig. 12. Once triggered, AEB aims to achieve acceleration a_AEB. In [61] a_AEB was assumed to be −10 m/s² on high friction roads. The build-up of deceleration is subject to a rate limit r_AEB as shown in Fig. 13. It should be noted that the AEB modeled here is an approximation but not necessarily a good representation of the actual AEB system on production vehicles.

Fig. 11.

Layout of the AV model.

Fig. 12.

TTC_AEB as a function of vehicle speed.

A first order lag with a time constant τ_AV is used to model the transfer function from the commanded acceleration to the actual acceleration for simplicity. The proposed accelerated evaluation process can be applied on other vehicle models such as CarSim [62], if more accurate simulations are desired.

The simulation parameters are listed in Table I.

TABLE I.

Parameters for the Lane-Change Simulations

Var. Unit	$T_{H{W}_d}^{ACC}$	$a_{ACC}^{\mathit{\text{Max}}}$	$K_p^{ACC}$	$K_i^{ACC}$	aAEB	rAEB	τAV	Ts	TLC
	S	m/s2	-	-	m/s2	m/s3	s	s	s
Value	2	5	−38.6	−1.35	10	−16	0.0796	0.1	8

B. Simulation Analysis

Three kinds of events were analyzed in this study:

• Conflict.

• Crash.

• Injury.

A conflict event happens when an AV appears in the proximity zone of the LCV between time t_LC and t_LC + T_LC. As shown in Fig. 14, the proximity zone is the area in the adjacent lane from 4 feet in front of the bumper of the LCV to 30 feet behind the rear bumper of the LCV [42, p. ix]. This area generally includes the blind spot and the area beside and behind the vehicle in which another vehicle is likely to travel.

Fig. 14.

Definition of conflict events.

The Cross Entropy is used to find optimal ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ and ${\vartheta }_{R_L^{-1}}$. The values of ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ and ${\vartheta }_{R_L^{-1}}$ in the tenth iteration are used in the simulations to estimate the probability of conflicts (conflict rate) in a lane change scenario. 100 lane changes are simulated in each iteration. As shown in Fig. 15, three sets of ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ and ${\vartheta }_{R_L^{-1}}$ are obtained with low, medium and high velocities. All ${\vartheta }_{R_L^{-1}}$ converge to about −0.12, whereas values of ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ float around zero. As the conflict events are defined based on R_L, R_L has a direct impact on the occurrence of the event. Therefore f̃_ϑ (x) is largely affected by $f_{R_L^{-1}}(x)$, and ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ converges to zero $({\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x){\tilde{f}}_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}(x))$.

Fig. 15.

Searching for optimal parameters for conflict events.

Both accelerated evaluation and the non-accelerated simulations (based on CMC) were conducted to demonstrate the performance and credibility of the proposed approach. Fig. 16 shows that the accelerated test is unbiased as the conflict rate converges to the one estimated in the non-accelerated simulation.

Fig. 16.

Estimation of the conflict rate.

The convergence is reached when the relative half-width l_r is below β = 0.2 with 80% confidence. Fig. 17 shows that the accelerated evaluation achieves this confidence level after N_acc = 364 simulations, while the naturalistic simulations take N_nature = 5.90e3 simulations.

Fig. 17.

Convergence of the conflict rate estimation.

In the SPMD database, during 1 325 964 miles naturalistic driving, 173 592 lane changes were identified with negative range rates. The frequency of lane change can be estimated as

$$r_{\mathrm{l}\mathrm{c}}=\frac{\mathrm{1\; 325\; 964}}{\mathrm{173\; 592}}=7.64[\text{mile}/\text{lane change}].$$ (51)

The driving distance needed in naturalistic test is thus

$$D_{\text{nature}}=r_{lc}·N_{\text{nature}}.$$ (52)

The test distance in accelerated evaluation

$$D_{\mathrm{A}\mathrm{C}\mathrm{C}}=\sum _{n=1}^{N_{\mathrm{A}\mathrm{C}\mathrm{C}}}\underset{t=t_{\mathrm{L}\mathrm{C}}}{\overset{t_{\mathrm{L}\mathrm{C}}+{𝒯}_{\mathrm{L}\mathrm{C}}}{\int }}{\upsilon }^{(n)}(t)dt$$ (53)

where υ⁽ⁿ⁾(t) represents the velocity of AV at time t in the n_th test and the termination time

$${𝒯}_{\mathrm{L}\mathrm{C}}=\text{min }\{\text{min }(t|R_L(t)<R_{ℰ}),T_{\mathrm{L}\mathrm{C}}\}.$$ (54)

The accelerated rate is defined as

$$r_{\text{acc}}=\frac{D_{\text{nature}}}{D_{\text{acc}}}.$$ (55)

The acceleration is achieved from both the modeling of lane change scenarios and the application of Importance Sampling and Cross Entropy techniques.

A crash happens when the range R_L becomes negative, i.e., R_ℇ = 0 in (24). Similar to the conflict events analysis, another Cross Entropy analysis is conducted to find optimal ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ and ${\vartheta }_{R_L^{-1}}$ for crash events. Because crashes are rarer than the conflict events, 500 lane changes are simulated in each iteration. As shown in Fig. 18, three different values of ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ were obtained from the iterative search for different velocity intervals, whereas ${\vartheta }_{R_L^{-1}}$ converges to values close to zero. It can be explained that in the crash analysis, the safety critical function (AEB) on AV is mainly affected by TTC. Therefore ${\vartheta }_{\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1}}$ has a larger impact on the occurrence of the crash. The estimation of the crash rate under accelerated and naturalistic conditions are shown in Fig. 19. The convergence is reached with 80% confidence level and β = 0.2 as shown in Fig. 20.

Fig. 18.

Searching for optimal parameters for crash events.

Fig. 19.

Estimation of the crash rate.

Fig. 20.

Convergence of the crash rate estimation.

Injuries are also important indicators of the performance of AVs. Here we focus on injuries with the Maximum Abbreviated Injury Score equal or larger than 2 (MAIS2+), representing moderate-to-fatal injuries. The probability of injury is related to the relative velocity at the crash time t_crash

$$\mathrm{\Delta }\upsilon =-{Ṙ}_L(t_{\text{crash}})>0.$$ (56)

The probability of moderate-to-fatal injuries for the AV passengers is estimated by a nonlinear model

$$P_{\text{inj}}(\mathrm{\Delta }\upsilon )=\{\begin{array}{cc}\frac{1}{1+e^{-({\beta }_0+{\beta }_1\mathrm{\Delta }\upsilon +{\beta }_2)}}\hfill & \text{crash}\hfill \\ 0\hfill & \text{no crash}\hfill \end{array}$$ (57)

which was proposed by Kusano and Gabler [63] shown in Fig.21 with parameters β₀ = −6.068, β₁ = 0.1, and β₂ = −0.6234. The injury rate E(P_inj(Δυ)) is calculated as

$$E(P_{\text{inj}}(\mathrm{\Delta }\upsilon ))={Ê}_{f*}(P_{\text{inj}}(\mathrm{\Delta }\upsilon ))\approx \frac{1}{n}\sum _{i=0}^{N_{\text{acc}}}{P}_{\text{inj}}(\mathrm{\Delta }\upsilon ({\mathit{x}}_n))L({\mathit{x}}_n)$$ (58)

where L is the likelihood and x_n represents the random variables $([{\upsilon }_L,\mathrm{T}\mathrm{T}{\mathrm{C}}_L^{-1},R_L^{-1}])$ in the nth simulation. The modified statistics used in crash events (shown in Fig. 18) are used to calculate the injury rate. The estimation results and convergence are shown in Figs. 22 and 23.

Fig. 21.

Moderate-to-fatal injury model for forward collisions.

Fig. 22.

Estimation of the injury rate.

Fig. 23.

Convergence of the injury rate estimation.

The accelerated rates of conflict, crash and injury events are summarized in Table II. The accelerated rates of crashes and injuries are higher than that of conflicts. This is because crashes and injuries occur with much lower probabilities than conflicts. The IS techniques generally have better performance when target events are rarer.

TABLE II.

Accelerated Rates of Conflicts, Crashes and Injury

	Conflict	Crash	Injury
Dnature [mile]	4.53e4	4.71e7	4.70e7
Dacc [mile]	16.4	4.02e3	2.53e3
racc	2.77e3	1.17e4	1.86e4

V. Conclusion

This paper proposes a new approach to evaluate the performance of AVs in an accelerated fashion. A lane change model was established based on a large naturalistic driving database—the Safety Pilot Model Deployment database. Lane change conflict, crash, and injury rates of a given AV model were estimated accurately but 2 000 to 20 000 times faster than the naturalistic driving tests in simulation. This technique thus has the potential to reduce greatly the development and validation time for AVs by providing both statistical conclusion and critical scenarios selected objectively.

In the future study, more comprehensive human-controlled model may be obtained as more data are collect in the Safety Pilot Model Deploy project and other projects. Other forms of IS distribution families other than ECM-based will be analyzed to potentially increase the evaluation efficiency to an even higher rate. The proposed accelerated evaluation approach can also be extended to other scenarios, such as car-following, lane departure or pedestrian avoidance and other testing platforms in addition to pure simulations, such that hardware-in-the-loop tests, driving simulator tests, or on-track tests.

Fig. 13.

Modeled AEB algorithm.

Biographies

Ding Zhao received the Ph.D. degree from University of Michigan, Ann Arbor, MI, USA, in 2016.

He is a Research Fellow with University of Michigan Transportation Research Institute, Ann Arbor, MI. His research interest includes automated vehicles, connected vehicles, driver modeling, and big data analysis.

Henry Lam received the B.S. degree in actuarial science from University of Hong Kong, Kowloon, Hong Kong, in 2005, and the A.M. and Ph.D. degrees in statistics from Harvard University, Cambridge, MA, USA, in 2006 and 2011, respectively.

From 2011 to 2014, he was an Assistant Professor with the Department of Mathematics and Statistics, Boston University, Boston, MA. Since 2015, he has been an Assistant Professor with the Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MA. His research focuses on stochastic simulation, risk analysis, and simulation optimization. His works have been funded by the National Science Foundation and the National Security Agency.

Dr. Lam is the recipient of an Honorable Mention Prize in the Institute for Operations Research and Management Sciences (INFORMS) George Nicholson Best Student Paper Award, and a Finalist in INFORMS Junior Faculty Interest Group Best Paper Competition.

Huei Peng received the Ph.D. degree from University of California, Berkeley, CA, USA, in 1992.

He is a Professor with the Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI, USA. He is the U.S. Director of the Clean Energy Research Center, Clean Vehicle Consortium, which supports 29 research projects related to the development and analysis of clean vehicles in the United States and in China. He also leads an education project funded by the Department of Energy to develop ten undergraduate and graduate courses, including three laboratory courses focusing on transportation electrification. He has more than 200 technical publications, including 85 in refereed journals and transactions. His research interests include adaptive control and optimal control, with emphasis on their applications to vehicular and transportation systems. His research focuses include design and control of hybrid vehicles and vehicle active safety systems.

Shan Bao received the B.E. and M.E. degrees in mechanical engineering from Hefei University of Technology, Hefei, China, and the Ph.D. degree in industrial engineering from the University of Iowa, Iowa City, IA, USA.

She is an Assistant Research Scientist with the Human Factors Group, University of Michigan Transportation Research Institute (UMTRI), Ann Arbor, MI, USA. She joined the UMTRI in 2009, starting as a Postdoctoral Fellow after completing her Ph.D. degree. Her research interests focus on driver behavior modeling, driver distraction, naturalistic driving data analysis, and driver-simulator study.

David J. LeBlanc received the Ph.D. degree in aerospace engineering from University of Michigan, Ann Arbor, MI, USA, and the bachelor’s and master’s degrees in mechanical engineering from Purdue University, West Lafayette, IN, USA.

Since 1999, he has been an Associate Research Scientist with University of Michigan Transportation Research Institute, Ann Arbor, MI. His work focuses on the automatic and human control of motor vehicles, particularly the design and evaluation of driver-assistance systems.

Kazutoshi Nobukawa received the B.E. and M.E. degrees in materials science and engineering from Waseda University, Tokyo, Japan, and the M.S.E. and Ph.D. degrees in mechanical engineering from University of Michigan, Ann Arbor, MI, USA.

From 2008 to 2010, he was a Graduate Student Research Assistant with University of Michigan Transportation Research Institute (UMTRI), Ann Arbor, MI. Since 2012, he has been a Research Fellow with the Engineering Systems Group, UMTRI. His research interest includes modeling and control of dynamical systems for analysis of collision avoidance systems, vehicle dynamics, tracking, and data mining.

Christopher S. Pan received the M.S. and Ph.D. degrees in industrial engineering from University of Cincinnati, Cincinnati, OH, USA, in 1989 and 1991, respectively.

He is a Senior Researcher with the National Institute for Occupational Safety and Health (NIOSH), Centers for Disease Control and Prevention, Morgantown, WV, USA. Since 1989, he has been conducting research with NIOSH, with projects chiefly focusing on ergonomics/safety. He currently has an appointment as an Adjunct Professor with the Department of Industrial and Management Systems Engineering, West Virginia University Morgantown, WV. He currently serves as a Project Officer with the NIOSH for six funded studies in construction and transportation sectors, including a follow-up collaborative project (2013–2015) of a motor vehicle study with University of Michigan Transportation Research Institute, Ann Arbor, MI, USA. For these and related research endeavors, he has been recognized by distinguished peers and professionals in the occupational safety and health community as a competent safety professional, project manager, ergonomist, inventor, and scientist.