Simulated Driving Assessment (SDA) for Teen Drivers: Results from a Validation Study

Abstract

Background

Driver error and inadequate skill are common critical reasons for novice teen driver crashes, yet few validated, standardized assessments of teen driving skills exist. The purpose of this study was to evaluate the construct and criterion validity of a newly developed Simulated Driving Assessment (SDA) for novice teen drivers.

Methods

The SDA's 35-minute simulated drive incorporates 22 variations of the most common teen driver crash configurations. Driving performance was compared for 21 inexperienced teens (age 16–17 years, provisional license ≤90 days) and 17 experienced adults (age 25–50 years, license ≥5 years, drove ≥100 miles per week, no collisions or moving violations ≤3 years). SDA driving performance (Error Score) was based on driving safety measures derived from simulator and eye-tracking data. Negative driving outcomes included simulated collisions or run-off-the-road incidents. A professional driving evaluator/instructor reviewed videos of SDA performance (DEI Score).

Results

The SDA demonstrated construct validity: 1.) Teens had a higher Error Score than adults (30 vs. 13, p=0.02); 2.) For each additional error committed, the relative risk of a participant's propensity for a simulated negative driving outcome increased by 8% (95% CI: 1.05–1.10, p<0.01). The SDA demonstrated criterion validity: Error Score was correlated with DEI Score (r=−0.66, p<0.001).

Conclusions

This study supports the concept of validated simulated driving tests like the SDA to assess novice driver skill in complex and hazardous driving scenarios. The SDA, as a standard protocol to evaluate teen driver performance, has the potential to facilitate screening and assessment of teen driving readiness and could be used to guide targeted skill training.

INTRODUCTION

In the United States in 2011, almost 2,000 drivers between the ages of 15 and 20 died in motor vehicle crashes (MVC) and an additional 180,000 young drivers suffered injuries, making MVCs the leading health threat to adolescents.(1) Poor driving skill, such as failure to recognize hazards, inattention to the forward roadway and driving too fast for conditions, contributes to nearly three-quarters of teen driver crashes.(2, 3) Even though we can identify major contributors, methods to reduce crashes among novice teen drivers have been hampered by a lack of valid assessments that measure driving skill, particularly in complex driving situations. Given that teen drivers account for a disproportionately high percentage of crashes (4) and the resultant morbidity and mortality exerts a societal burden (5), validated, standardized assessment of teen driving skill is needed.

On-the-road assessments of driving performance with evaluators (6), while important and necessary, are limited in hazard exposure and include threats to validity. Key factors, such as traffic and environmental conditions, cannot be replicated and study designs cannot place teens in dangerous on-the-road driving situations. However, teens crash in complex and dangerous scenarios (7, 8); therefore, assessment of how teens perform in these situations can provide important information for prevention strategies. Driving simulators could offer a replicable, safe complement to on-the-road evaluations (9); however, no validated driving simulator protocols to assess teen driver performance exist. Our goal was to create a standardized, replicable simulated assessment of teen driving performance in scenarios known to be associated with teen crashes, delivered within a driving simulator's safe environment.

We have undertaken a series of systematic steps to develop an assessment of teen driver performance: the Simulated Driving Assessment (SDA). The scenarios and measurement strategies in the SDA were theoretically grounded in actual crash data. The SDA was designed to expose drivers to complex driving scenarios that are often the cause of serious crashes and to measure their driving performance via key metrics determined from the empirical literature and expert opinion.(10–17) Crash data from the National Motor Vehicle Crash Causation Survey (NMVCCS) were used to determine the SDA simulated scenarios.(18, 19) The top three types of serious crashes (rear-end collisions, left turn intersection collisions, and right side run-off the road events) represented approximately 30% of NMVCCS teen crash configurations.(19) Variations of these three types of serious crashes became the driving scenarios in the SDA so that the assessment would approximate the common real world scenarios in which teen drivers crash; i.e., have ecological validity. (19) Initial analyses were carried out to help establish the utility of the SDA. We examined the potential for learning effects during the SDA and found no training effect or induced hypervigilance because of exposure to multiple potential crash scenarios in the SDA.(20) Given that a driving evaluator instructor (DEI) often serves as the best available gold standard for assessing driving performance, we compared DEI rated drivers' performance during the SDA (a derived categorization of skill/less skilled) to history of police reported crashes. Seven of the eight (87.5%) experienced adult drivers who had at least one police-reported crash were categorized as less skilled by the DEI, resulting in a sensitivity of 0.87 (95% CI: 0.47–0.99). (21)

Given this foundation, the purpose of this study was to assess the construct and criterion validity of the SDA for novice teen drivers, crucial tests of its utility for driver assessment. In order to evaluate the construct validity of the SDA, we: 1.) compared safe driving performance measures among teens and adults (Error Score) and; 2.) assessed the ability of the SDA to predict Simulated Negative Driving Outcomes from the Error Score. In order to evaluate the SDA's criterion validity, we compared the Error Score to an expert criterion, an independent video review of SDA performance by a professional driving evaluator/instructor (DEI Score).

METHODS

Simulated Driving Assessment (SDA) Procedures

The methods to develop the SDA scenarios have been described above. Prior to completing the SDA, participants drove an unscored familiarization segment (~7–10 minutes) to adjust to simulator dynamics. The SDA lasted approximately 35–40 minutes and included three modules that were randomized (by a random number generator). During the SDA, participants were exposed to 22 variations of the most common teen driver crash configurations (rear-end collisions, left turn intersection collisions, and right side run-off the road events).(19) The 22 crash scenarios were distributed across the three modules, separated by intervening straight roads, curves and turns not intended to trigger collisions. The length between scenarios ranged between about 300–1800 meters, with an average of about 930 meters. If the participant drove the SDA safely, crashes were avoidable. In order to decrease the potential for a learning effect, no feedback on performance was given to participants on their performance during or after the drive.(20)

Simulated Driving Assessment (SDA) Performance Metrics

The 14 driving performance metrics scored during the SDA (determined by the empirical literature and expert opinion) represent key measurable driving behaviors, including basic (e.g. turn signal use) and advanced (e.g. presence of hazard anticipation glances) maneuvers. Table 1 outlines the definitions, calculations, sources of data and criteria for errors in each of the metrics, all of which were determined by on-the-road and/or simulator studies. Table 2 outlines the potential crash scenarios, number of occurrences in the SDA and errors (dichotomous) scored in the 22 potential crash scenarios.

Table 1.

Driving Performance Metric Definitions, Calculations and Criterion for Error

Driving Performance Metric	Definition of Metric	Calculation	Source	Criterion for Error in SDA
Operational Maneuvers
Lane choice (10)	Lane selection during turn at an intersection	Lane position up to 2 seconds prior to initiation of turn as indicated by a turn of the steering wheel beyond +/− 45 degrees	Simulator	Controlled and uncontrolled left turns: Turning left at intersection from the right lane.
Lateral acceleration during turn (11)	Lateral acceleration during a turn at an intersection	Lateral acceleration	Simulator	Controlled and uncontrolled left turns: Lateral acceleration ≥0.5 g's.
Speed (10)	Velocity	Velocity relative to posted speed limit	Simulator	Controlled left turns, rear-end and hidden hazard (car in driveway): Velocity exceeds posted speed limit by 10% in clear conditions and 5% in rainy conditions. Curves: Velocity exceeds posted speed limit at any point in the curve. Uncontrolled left turns: Velocity exceeds posted speed limit at any point in the turn.
Stopping behavior at a stop sign (10)	Velocity at a stop sign	Minimum velocity at stop sign	Simulator	Controlled left turns: Failure to reach velocity of zero at a stop sign.
Traffic check (10)	Glance sequence at intersection	Coding of glance location sequence	Eye tracking data-video	Controlled left turns: Failure to make a glance sequence of left-right-left within 5.5 meters of intersection.
Turn signal use (10)	Application of turn signal to alert other road users of a turn	Activation of turn signal	Simulator	Controlled and uncontrolled left turns: Failure to use turn signal.
Yaw (11)	Vehicle heading correction after completion of a turn at an intersection	Vehicle heading correction after completion of turn as indicated by the return of the steering wheel to 0 °	Simulator	Controlled and uncontrolled left turns: Vehicle heading correction is altered by more than ± 6° after turn completion.
Advanced Maneuvers
Braking behavior in a hazardous situation (11–13, 16)	Instance of substantial longitudinal deceleration in response to a hazardous situation	Rear-end events: Based on headway time and instance of longitudinal deceleration >0.6g prior to a collision Hidden hazard (car in driveway): Based on velocity and instance of longitudinal deceleration >0.6g prior to a collision	Simulator	Rear-end: Headway time <3 seconds and no instance of longitudinal deceleration >0.6g. Hidden hazard (car in driveway): Velocity exceeds posted speed limit by 5% and no instance of longitudinal deceleration >0.6g.
Excess Glance Duration>1.5 seconds (14)	Summed glance duration off forward roadway over 1.5 seconds	Time duration between when the crosshairs of the ASL eye-tracker left the area of the center display view and reappeared during a distraction task: duration of glances were summed	Eye tracking data-video	Rear-end with distraction: Glance duration off forward roadway > 1.5 seconds.
Gap selection (17)	Choice of time to enter an intersection and proximity to other vehicle in intersection	Video coding determined if participant waited for cross traffic. For participants that did not wait, post-encroachment time (PET) was calculated	Simulator (video and data)	Controlled left turns: PET<1.5 sec. Participants who waited for cross traffic did not receive an error.
Headway time (16)	Instantaneous time to reach lead vehicle	Distance/velocity of participant vehicle center of gravity to lead vehicle center of gravity. Video coder estimated distance for a minority of participant scenarios where vehicle information was not available.a	Simulator	Rear-end: Distance/velocity of participant vehicle to lead vehicle <3 seconds.
Scan- left for intersection (15)	Glance to the left as participant approaches an intersection	Video coder identified glances to left as participant approached intersection: Coded from the location of instructions to make a left turn until ~ 5.5 meters from intersection	Eye tracking data-video	Controlled and uncontrolled left turns: No recorded glances to left as participant approached intersection.
Scan- right for intersection (15)	Glance to the right as participant approaches an intersection	Video coder identified glances to right as participant approached intersection: Coded from the location of instructions to make a left turn until ~ 5.5 meters from intersection	Eye tracking data-video	Controlled and uncontrolled left turns: No recorded glances to right as participant approached intersection.
Scan-right for hidden hazard (15)	Glance to the right as participant approaches a hidden hazard	Coder identified glances on fence, driveway and car as participant approached hazard event and record time stamps of glances and when car pulls out.	Eye tracking data-video	Hidden hazard (car in driveway): No recorded glances to right as participant approached hidden hazard event.

^aVideo coding calculation of headway time to known simulator derived values of headway time and found that the mean difference between both samples was 0.13 seconds (95% CI: 0.01–0.27)

Table 2.

Scoring of Driving Performance Metrics in Potential Crash Scenario Type

Potential Crash Scenario Type	Variations	Number in SDA	Driving Performance Metric
			Lane choice	Lateral acceleration during a turn	Speed	Stopping behavior at a stop sign	Traffic check	Turn signal	Yaw	Braking behavior in a hazardous situation	Excess glance duration	Gap selection	Headway time	Scan-left for intersection	Scan-right for intersection	Scan-right hidden hazard
Controlled left turns	3-way intersections with cross traffic, no obstruction	2	X	X	X	X	X	X	X			X		X	X
	3-way intersection with cross traffic, with an obstruction	2	X	X	X	X	X	X	X			X		X	X
	4-way intersections with cross traffic, no obstruction	2	X	X	X	X	X	X	X			X		X	X
	Out of a parking lot	2		X	X		X	X	X			X
Curves	Curves in rainy weather	3			X
Rear-end – no distraction	Lead vehicle stops suddenly in the left lane in order to turn left	2			X					X			X
	Lead truck in right lane stops suddenly	3			X					X			X
Rear-end – with distraction	Lead truck in right lane stops suddenly while participant looks for loose change or tunes the radio	2			X					X	X a		X
Hidden hazard (car in driveway)	Car emerges from the right from an obstructed driveway in rainy weather	2			X					X						X
Uncontrolled left turns	3-way intersection where participant has a green light	2	X	X	X			X	X					X	X

^aExcess glance duration was only scored in 1 distraction event

Apparatus

The Realtime Technology, Inc. (RTI)® fixed-based driving simulator system used in this study included a driver seat, three-channel 46” LCD panels (160° field of view), rearview, left and right mirror inlayed images, active pedals and a steering system. Visual rendering and graphics were delivered at 1280 × 1024 resolution at 60-Hz. Raw simulator data (e.g. velocity, position) were collected at 60 Hz and were reduced to the 14 driving performance metrics as outlined in Table 1. Videos of participant's driving behavior in the simulator were also recorded by three cameras (positioned over the right shoulder, pointing at the participant's face and over the foot for view of brake and accelerator). Video playback data were assessed by the DEI (further described below).

Applied Science Laboratories (ASL) Mobile Eye® was used to capture eye movements on the forward scene at 30Hz. Participants wore a pair of glasses (or alternative goggles worn over eyeglasses) with two video cameras (forward image and right eye of participant) to capture gaze location during the SDA. Video data from the two cameras were integrated into a single video with superimposed cross hairs for eye gaze location. Videos were used for coding in analysis as outlined in Table 1.

Participants

Two groups were enrolled: 1.) Inexperienced teens (age 16–17 years, Pennsylvania (PA) provisional license ≤90 days) and 2.) Experienced adults (age 25–50 years, PA license ≥5 years, drove ≥100 miles per week, no collisions or moving violations ≤3 years).

Individuals were excluded for a self-report history of migraines or motion sickness, current pregnancy or non-English speaking. Teen participants were recruited via mailings from pediatric primary care facilities, driving schools, and word of mouth. Experienced adults were recruited via flyers, the Wharton Behavioral Lab at the University of Pennsylvania, and word of mouth. Participants were compensated $50. Study procedures were approved by the Institutional Review Boards at The Children's Hospital of Philadelphia and an administrative agreement with the University of Pennsylvania.

Main Measures

Demographic data

Self-report data on age, gender, race/ethnicity and length of licensure were collected.

Error Score

Error Score was based on errors in 14 driving performance metrics in the 22 potential crash scenarios. Only errors relevant to a specific scenario were calculated (e.g. turning from a correct lane was not scored in the rear-end events; see Table 1 for calculations and Table 2 for scoring). A minimum of 1 and maximum of 10 driving performance metrics were scored in each of the 22 scenarios. A sum score of errors in driving performance metrics was aggregated across potential crash scenarios (Error Score). Potential range of the Error Score was 0–117.

Simulated Negative Driving Outcome (Simulated Collisions and Run-off-the-Road)

Each of the 22 scenarios had potential for a negative driving outcome, either a simulated collision or run-off-the-road incident. Only one Simulated Negative Driving Outcome could be assigned per scenario, whichever occurred first. Simulated collisions were defined as an overlap of the participant's vehicle with other vehicles programmed for the potential crash scenario. Collisions were derived from simulator data on position, orientation, and dimensions of the participant and nearest vehicle and were verified by video review. For five participants, automated simulator data for collisions were unavailable due to a programming error and were independently video coded and reconciled for 100% agreement by two research team members. Run-off-the-road incidents were defined as any instance of the vehicle perimeter departing the roadway (either side). Run-off-the-road incidents were derived from simulator data of participant vehicle position, orientation, and dimensions. Possible range of total SDA Simulated Negative Driving Outcomes was 0–22.

Driving Evaluator/Instructor (DEI) Score

A professional DEI with 20 years of driver training experience reviewed a four-quadrant divided screen of SDA video playback of the three scored modules (familiarization segment excluded). The video included forward roadway, right foot behavior (brake and throttle), and over the participant's right shoulder and face. Speed in miles per hour (mph) and left and right turn signals were displayed; however, no simulator or eye-tracking data or selected measures for scoring were provided to the DEI. The DEI was blinded to driver age and SDA Error Score. The DEI rated driving skill (scale 0–10; 0 =unskilled and 10 = proficient) in eight domains: attention maintenance, communication and right of way, gap selection, hazard anticipation and response, managing blind spot, road positioning, speed management, and vehicle control. As previously stated, this DEI demonstrated a high sensitivity in identifying skill level in the SDA as related to previous crash history.(21) A sum score was created (DEI Score) with a possible range of 0–80.

Analysis

Table 1 outlines metric definitions, calculations, sources of data and criteria for errors. Custom MATLAB (Mathworks, Inc., Natick, MA) code was used to reduce raw simulator data. For categorical variables, frequencies and percentages were computed. For continuous variables, median, interquartile range [IQR], and full range were computed. Construct validity was examined by assessing the following hypotheses: (H₁) Teen drivers would have a higher Error Score than adults; and (H₂) higher Error Score would predict increased propensity for a Simulated Negative Driving Outcome. Wilcoxon Rank Sum Test and Fisher's Exact Test were used to compare teen and adult performance. A modified Poisson regression was used to estimate the relative risk (RR) and 95% confidence interval (CI) to predict the total sum of Simulated Negative Driving Outcomes based on experience (inexperienced teen or experienced adult) and Error Score. Criterion validity was examined by assessing the following hypothesis: (H₃) DEI score would be negatively associated with Error Score (e.g. participants rated as more proficient by the DEI would have fewer errors-or a lower Error Score). Spearman correlation was used to determine the association between Error Score and DEI Score. The videos viewed by the DEI included obvious collisions and run-off-the-road instances; therefore the association between DEI score and negative driving outcomes was not examined. For hypothesis testing, the Type-I error rate was set at α=0.05. All statistical analyses were performed using R v3.0.1 (www.r-project.org).

RESULTS

Our analytic sample included 38 participants who completed the SDA (see Figure 1). The teen group included 21 participants: 33% female, 95.2% White, 4.8% African American, 100% non-Hispanic, median age=16 years (IQR16–17; range16–17), with provisional license for a median of 35 days (IQR 7–44; range 1–88). The adult group included 17 participants: 41% female, 70.6% White, 23.5% African American, 5.9% race not reported, 100% non-Hispanic, median age=34 years (IQR 28–42; range 25–50), licensed for a median of 15 years (IQR 11–22; range 8–33). There were no statistically significant differences between the teen and adult groups across sex, race, and ethnicity. As expected, statistically significant differences between the teens and adults were observed for age (Z=5.40, p<0.001) and length of licensure (Z=5.23, p<0.001) (Wilcoxon Rank Sum Tests).

Figure 1.

Derivation of Analytic Sample

Table 3 includes the SDA Error Score and errors in each driving performance metric for teens, adults and the total sample. For the total sample, the Error Score range was 4 to 54 errors (teen range, 13–54, adult range, 4–50). Consistent with H₁, teens had a higher Error Score than did the adults (Z= −2.37, p=0.02 Wilcoxon Rank Sum Test). Note, median and IQRs of errors and frequency of teens and adults with ≥1 error in each driving performance metric during the entire SDA are also reported in Table 3; however given the small sample, we consider comparisons between teens and adults exploratory and thus do not report p-values.

Table 3.

Error Score, Errors in each Driving Performance Metric and Simulated Negative Driving Outcomes

		Total Sample (n=38)	Teens (n=21)		Adults (n=17)		P value
	Possible range	Median [IQR]	Median [IQR]	Teens with ≥1 Error in each Driving Performance Metric n (%)	Median [IQR]	Adults with ≥1 Error in each Driving Performance Metric n (%)
Error Score	0–117	22.5 (13–36)	30 (19–36)		13 (10–32)		0.02*
Driving Performance Metrics
Lane choice	0–8	0 (0–0)	0 (0–0)	5 (23.8)	0 (0–0)	1 (5.9)
Lateral acceleration during a turn	0–10	2 (0–6)	2 (1–6)	16 (76.2)	2 (0–5)	11 (64.7)
Speeding	0–22	4 (2–7)	6 (2–7)	19 (90.5)	3 (0–4)	12 (70.6)
Stopping behavior at a stop sign	0–6	0 (0–1)	0 (0–3)	8 (38.1)	0 (0–1)	7 (41.2)
Traffic check	0–8	3 (1–4)	3 (2–4)	14 (66.7)	2 (0–3)	16 (94.1)
Turn signal	0–10	0 (0–0)	0 (0–0)	0 (0)	0 (0–0)	3 (17.6)
Yaw	0–10	1 (0–1)	0 (0–1)	11 (52.4)	1 (0–2)	12 (70.6)
Braking behavior in a hazardous situation	0–9	2 (1–3)	3 (2–4)	21 (100)	1 (0–2)	12 (70.6)
Excess glance duration	0–1	0 (0–0)	0 (0–1)	6 (28.6)	0 (0–0)	3 (17.6)
Gap Selection	0–8	6 (4–8)	6 (5–8)	8 (38.1)	6 (4–7)	8 (47.1)
Headway time	0–7	3 (2–5)	4 (3–5)	21 (100)	2 (1–3)	13 (76.5)
Scan-left for intersection	0–8	3 (1–4)	4 (3–5)	20 (95.2)	1 (0–3)	15 (88.2)
Scan-right for intersection	0–8	3 (2–5)	4 (2–5)	21 (100)	3 (2–4)	11 (64.7)
Scan-hidden hazard	0–2	0 (0–0)	0 (0–1)	7 (33.3)	0 (0–0)	2 (11.8)
Simulated Negative Driving Outcomes	0–22	0 (0–2)	0 (0–2)		0 (0–1)		0.43

Note: Error Score is the sum score of errors in driving performance metrics aggregated across potential crash scenarios. Teens and adults had similar distributions of individual errors, error score and simulated negative driving outcomes. Missing Data: For a minority of cases with device calibration failure, eye-tracker-derived measures could not be calculated for the hazard anticipation glances, traffic check and summed excess glance duration (missing data for each measure by scenario ranged from 0–21%). Missing data for hazard anticipation glance or traffic check data were imputed from available data in the other scenarios. Given that there was only one event scored for summed excess glance error, missing data for this driving performance error were assigned a non-error.

During the SDA, 42.9% of teens and 29.4% of adults had at least one Simulated Negative Driving Outcome. Total sample range was 0 to 4 (teen range 0–3 and adult range 0–4); total sample median was 0 [IQR 0–2] (teens median 0 [IQR 0–2]; adult median 0 [IQR 0–1)]). There were no statistically significant differences in total Simulated Negative Driving Outcomes (Z= −0.80, p=0.43 Wilcoxon Rank Sum Test) between teens and adults or percentage of teens and adults with at least one Simulated Negative Driving Outcome (p=.51, Fisher's Exact Test). Note: among negative driving outcomes, 94% were simulated collisions.

Consistent with H₂, across all participants, for every one additional driving error, there was an 8% increase in the risk of a simulated negative driving outcome (RR=1.08; 95% CI, 1.05–1.10, p<.01). Figure 2 plots the exponential relationship between the propensity to have a Simulated Negative Driving Outcome (mean of lambda) and Error Score. Using a Wald Test of the coefficients (Z= −0.36, p=.72) and Akaike Information Criteria (AIC), experience was rejected as a covariate. Experience alone was not a significant predictor of Simulated Negative Driving Outcome (RR=1.38; 95% CI, 0.67–3.04).

Figure 2.

Mean number of simulated negative driving outcomes vs SDA Error Score.

DEI score ranged from 6 to 66 (teen range 23–66; adult range, 6–65) with a median of 46 and IQR 32–59 (teen median=46, IQR 34–58; adult median=46; IQR 32–61). There was no statistically significant relationship between DEI score and experience (Z= −0.25, p = 0.80). However, consistent with H₃, Error Score was significantly associated with DEI Score (r=−0.66, p<0.001).

DISCUSSION

The results provide support for the construct and criterion validity of the SDA in its ability to differentiate driver performance by use of established safety metrics and to predict simulated negative driving outcomes. Teen drivers within three months of licensure made more total errors in the SDA than did experienced adult drivers. Further, the SDA demonstrated a range in driver performance errors and a near linear relationship between Error Score and propensity to have at least one Simulated Negative Driving Outcome; with each error in the SDA, the risk for crashing or running off the road increased 8% (RR=1.08; 95% CI, 1.05–1.10, p<.01). Evidence for criterion validity is provided by the association between independent expert rating of driving performance and calculated scores of safety measures derived from simulator and eye-tracker data (r=−0.66, p<0.001).

The SDA Error Score, but not ratings from expert video review, showed a difference in driving performance between teen and adult drivers. The lack of association of video review and experience was likely due to potential limitations in video review for assessment of performance. Even though a DEI is often considered the best available standard for evaluation, the position of the DEI in a vehicle may limit ability to fully assess skill. For example, from the video review, the DEI was not able to precisely see where the participants' were looking (i.e. eye glance location) which is similar to the situation with on-road assessments where an evaluator's view may be restricted. Given that poor scanning contributes to novice driver crashes (2, 3), a simulator and eye tracker can play an important role in assessing crucial skills like hazard anticipation and glance duration off the forward roadway.

A key goal of the SDA is to differentiate between skilled and non-skilled drivers; thus we used experience as a proxy for skill. Although adults in our sample were experienced, they did not necessarily perform like skilled drivers. This is consistent with our previous research which revealed that the experienced adults, while eligible for the study, may have included drivers who were less skilled and less safe drivers: 8 of the 17 adults had at least one police-reported crash during their PA licensure, none that occurred during the 3-year window used in the study exclusion criteria.(21) (Note: none of the inexperienced group had a police-reported crash prior to enrollment in the study). Therefore, our adult driver group contained some drivers who did not demonstrate the “gold standard” for safe driving. This may, in part, explain why a statistical difference between the adult and teen group was found in Error Score, but we did not find that the SDA differentiated Simulated Negative Driving Outcomes based on experience. There was a trend, however, to suggest that teens may have more Simulated Negative Driving Outcomes than adults (42.9% vs. 29.4%; RR=1.38, 95% CI: 0.67–3.04, p>.05). With the narrow distribution of the continuous count of crashes and dichotomous outcome variable (no crashes versus ≥1 crashes), analysis of crash counts was likely underpowered to detect statistical differences. Future validation of the SDA (and other studies that utilize adults as a skilled/safe driver comparison group) may need to use with a larger sample and additional criteria that enhance exclusion of unskilled, unsafe drivers.

Some of our individual driving performance metrics had a median of 0 errors by teens or adults (e.g. turn signal, lane choice). These metrics were included to provide face validity of the new SDA tool as these are common metrics used by driving assessors as part of licensing. However, future iterations of the SDA scoring may choose to refine or eliminate these metrics due to lack of variation among participants. Other metrics proved more variable, and demonstrated not only the utility of the tool but also the range of driving skill among the participants. In addition, future research with a larger sample should explore how each error metric differs among teens and adults.

By examining multiple performance measures in scenarios designed to replicate the most common crash configurations, the SDA provides information about skill deficits in hazardous situations. These findings demonstrate the SDA's utility as a measure of driving skills known to be involved in teen driver crash risk. Teens displayed mastery of basic skills, such as turn signal use, but more advanced driving behaviors proved challenging. Teens had repeated errors of advanced behaviors, including braking in hazardous situations and hazard anticipation glances and demonstrated repeated negative driving outcomes during the SDA. These results complement those from an evaluation of an on-road assessment demonstrating teens' ability to perform basic maneuvers but deficits in more advanced, critical driving behaviors.(22)

This study is not without limitations. The simulated driving environment replicates but is not identical to on-the-road driving; however, the SDA's advantage is to safely expose teens to high risk driving situations. Future prospective studies with larger samples should examine the ability of the SDA to predict teen driver on-road behavior and crashes. The study inclusion/exclusion criteria limit the applicability of the results to the populations that were studied. Larger samples are needed to generalize results to the broader population or to special populations with medical and psychiatric conditions. The metrics for scanning (e.g. scan left for intersection) were not an exact replication of scenarios and measurement of the hazard anticipation glances in the Fisher et al.¹⁵ work. However, the emphasis of this metric on scanning in potentially hazardous situations was the basis for selection. Future work should further examine measurement strategies for scanning in the SDA. Incomplete performance data were available for 15 adults and one teen that stopped the SDA due to simulator sickness (see Figure 1 for determination of sample). Overall, the percentage of adults with simulator sickness in our study (36.6%) is consistent with reports in the literature.(12) For the stated purpose of the SDA – assessment of novice teen driver performance – the SDA resulted in minimal simulator sickness. This study focused on assessment of driving skill in relationship to experience as a marker of safe driving. A more comprehensive assessment of driving risk should also include risky driving and behavioral tendencies. A single rater (DEI) with over 20 years of experience in driver training was used to assess video recordings, of which was only one form of validity. In the future, the use of multiple raters might be of value although we confirmed the ability of this DEI to accurately identify unskilled, unsafe drivers.(21)

CONCLUSIONS

This study supports the validity of the SDA for use in a safe and replicable manner to assess novice driver skill in complex and hazardous driving scenarios. The SDA, as a standard protocol to assess teen driver performance, could facilitate screening and assessment of teen driving readiness and could be used to guide targeted skill training.

What's Known on This Subject

• Driver error and inadequate skill are the main contributors to teen driver crashes.

• On-road assessments of driving performance are limited due to limited exposure (safety concerns) and replicability (variable traffic/environmental conditions).

• There is a gap in validated assessments of teen driver skill, particularly in complex driving situations which have inherent risk.

• Advanced driving simulators provide a realistic yet safe complement to on-road tests.

What This Study Adds

• This study demonstrated validity of the Simulated Driving Assessment (SDA) for safe evaluation of skill in high risk driving scenarios.

• A tool like the SDA could inform teen driving evaluations and guide individualized driver training and management plans.