Road Test and Naturalistic Driving Performance in Healthy and Cognitively Impaired Older Adults: Does Environment Matter?

Abstract

Background/Objectives

The road test is regarded as the gold standard for determining driving competence in older adults, but it is unclear how well the road test relates to naturalistic driving. The study objective was to relate the standardized road test to video recordings of naturalistic driving in older adults with a range of cognitive impairment.

Design

Cross-sectional observational study.

Setting

Academic medical center memory disorders clinic.

Participants

103 older drivers (44 healthy and 59 with cognitive impairment) who passed a road test.

Measurements

Error rate and global ratings of safety (pass with and without recommendations, marginal with restrictions or training, or fail) made by a professional driving instructor.

Results

There was fair agreement between global ratings on the road test and naturalistic driving. More errors were detected in the naturalistic environment, but this did not impact global ratings. Error scores between settings were significantly correlated, and the types of errors made were similar. History of crashes corrected for miles driven per week was related to road test error scores, but not naturalistic driving error scores. Global cognition (MMSE) was correlated with both road test and naturalistic driving errors. In the healthy older adults, younger age was correlated with fewer errors on the road test and greater errors in naturalistic driving.

Conclusion

Road test performance is a reasonable proxy for estimating fitness to drive in older individuals’ typical driving environments. The differences between performance assessed by these two methods, however, remain poorly understood and deserve further study.

Introduction

The road test is widely accepted as the gold standard for licensing new drivers. Given its face validity, it has also been adapted as a tool to monitor older adults for potential declines in driving ability related to physical illness or cognitive impairment¹. Test characteristics for on-road tests in cognitively healthy older adults show good inter-rater and test-retest reliability, internal consistency, and correlation between global ratings and performance scores in research settings ^2,3. Road test performance is also related to office-based cognitive assessment ^4-10 and history of motor vehicle crashes ¹¹.

Despite this, little is known about the relationship between road test and naturalistic driving behavior. Although the road test occurs in a real-world environment, there are environmental differences between the two settings that may impact its ecological validity. For example, the road test may occur in an unfamiliar or more complex environment. It is well-documented that some older adults with and without cognitive impairment restrict their driving space and may reduce the complexity of their driving environments (e.g., avoid nighttime and highway driving). ^12-14 As such, placing the participant in a new and possibly more difficult course may place additional cognitive demands on the participant. Formal test-taking with potential consequences may also lead to anxiety. In a large study of road test performance in healthy and cognitively impaired older adults, road testing was associated with test taking fear and anxiety in some participants ¹⁵. In a retrospective review of road test performance of older adults, road test failures substantially increased when individuals were no longer allowed to use their own vehicle during the road test¹⁶. Older adults may therefore be more susceptible to the negative influences of anxiety and course and vehicle unfamiliarity during the road test. These factors may in turn elicit driving difficulties that are not apparent in their typical environments, resulting in inappropriate failures of individuals who are safe to drive. Conversely, some drivers may be on their “best behavior” while being monitored by an in-car instructor. Moreover, older adults may benefit from being accompanied by a driving instructor who is providing navigation instructions and cueing regarding traffic situations or driving rules. In fact, prior research suggests that cueing enhances performance on the road test ¹⁷.

To date, there are no accepted standards for the road test for older drivers ^18,19. This is in part because our knowledge about older adults’ driving behavior is largely derived from self- or caregiver report and not direct observation ²⁰. Based on survey data, older adults limit their driving to well-known streets, reduce their weekly mileage, and reduce the complexity of their driving by avoiding highways and night driving ^{14, 21-23}. Recent naturalistic data using in-car recording devices confirms these self-report findings and demonstrates that driving behavior declines when an older adult is faced with challenging driving situations and is driving in environments that are further away from home ²⁴. When driving was monitored over one week, older drivers had fewer demerit points for unsafe driving, but made different types of errors, including not stopping at stop signs and turning errors, compared to younger participants ²⁵.

The validity of the road test in predicting the types of errors and problems detected with naturalistic methods is largely unknown given the paucity of studies directly comparing road test to naturalistic driving performance. There are many environmental differences between these two settings, and factor analysis of a standardized road test compared to naturalistic driving suggests that the road test requires a more limited set of skills than naturalistic driving, raising concern that road tests are not optimally designed to estimate driving competence in older adults²⁶. Furthermore, it is unclear if the structured and cued format to the road test enhances driving performance or reveals problems that would not otherwise have been identified. The objective of this study was to compare road test to 4 hours of video recorded naturalistic driving in older adults with a range of cognitive impairment.

METHODS

Participants

Study participants were ages 60-90, with a valid driver’s license, and no at-fault crashes within the past year. Physician or family member concern about driving competence was not required for study entry. Healthy older adults had no history of dementia and a Mini Mental State Examination (MMSE) ²⁷ score greater than 26. Cognitively impaired participants were recruited primarily from a hospital-based memory disorders clinic during routine follow-up visits for their memory disorder. Healthy participants were either spouses of patients or were recruited through community advertisements. Dementia severity was measured with the Clinical Dementia Rating scale (CDR; ²⁸ only patients with CDR scores of.5 or 1 were included. Patients met diagnostic criteria for possible or probable Alzheimer’s disease (AD) based on the NINCDS-ADRDA criteria ²⁹. Patients were on a stable dose of a cholinesterase inhibitor for six weeks, if prescribed. Study informants were individuals who accompanied the participant while driving at least once monthly during the preceding year and could report on driving history. Study informants provided information on crash history and miles driven for the memory disordered participants, and they also were available to drive the participant home if they failed the on-road test. Exclusion criteria for both groups included reversible causes of dementia, physical or ophthalmologic disorders that might impair driving abilities, mental retardation, schizophrenia, bipolar disorder, or alcohol/substance abuse within the previous year. Anxiolytic and antipsychotic medications were permitted, but dosages were required to be stable for 6 weeks before study entry.

Procedure

Written informed consent was obtained from all participants. Rhode Island Hospital Institutional Review Board approved all procedures. Participants were screened for inclusion criteria at an office visit that included the neurological examination, vision screen, MMSE, and CDR. Participants were scheduled for a road test if inclusion criteria were met. If they failed the road test, they could re-test within 2 weeks, but they were encouraged to refrain from driving during those 2 weeks until the re-test was completed. Of the initial 122 participants enrolled, 10 failed the road test (CDR=0, n=1; CDR=.5, n=5; CDR=1, n=4). One participant re-tested and passed. Cameras were then installed into their primary vehicle to record naturalistic driving. Participants who failed the road test or elected not to re-test were terminated from the study due to ethical considerations related to video recording drivers who could not pass a road test. A letter recommending driving cessation was sent to the primary care physician, and participants and family members were counseled by the driving instructor.

Miles Driven and Crash History

Miles driven per week and crash history were obtained at the baseline visit by the informant for the memory disordered participants and self-report by the healthy controls. Both types of information were obtained in a semi-structured interview format by the research assistant.

Standardized Road Test

Participants were administered the Rhode Island Road Test (RIRT) by a professional driving instructor (who was blind to diagnosis). The same driving instructor rated all participants for the duration of the study. The RIRT was administered within one month of the baseline office assessment during daylight hours under good road conditions. The test covered 6.5 miles of urban terrain without highway driving and required 45 minutes to complete. The driving instructor accompanied the participant in a specially fitted vehicle that had a brake on the passenger side for emergency use. A pretest of basic vehicle operation was performed in a parking lot prior to the test. The driving instructor only provided verbal instructions to complete the course.

The RIRT is an adaptation of the Washington University Road Test, a standardized driving measure with previously established reliability ³⁰. The RIRT showed adequate inter-rater reliability for 20 participants (rated by a second professional driving instructor in the back seat) with almost perfect agreement for the global rating (kappa 0.83 for linear weighted ratings and 0.92 for quadratic weighted ratings) ^{31, 32} and a strong correlation coefficient for the error score 0.87 ³³.

Twenty-eight driving maneuvers/behaviors were rated on a 3-point scale (0= unimpaired, 1=mildly impaired, 2=moderately to severely impaired) (see ²⁶ for complete description). The majority of participants had the same number of opportunities to engage in a driving maneuver across the test. If an event was not encountered (e.g., response to pedestrian), it was not included in the overall proportional score. Total scores ranged from 0 to 960, with higher scores reflecting poorer performance. Two dependent variables were generated. The first variable was the average error severity score, reflecting the sum of ratings (range 0-2) for each event divided by the number of observed maneuvers. The second variable was a categorical rating of overall driving ability (Pass with or without Recommendations, Marginal with Restrictions, Marginal with Training, or Fail). Pass implied that the participant’s driving performance would unlikely result in crashes or violations. Recommendations included behaviors that could improve safety. Marginal indicated that the driver could continue to drive, but should restrict driving to particular locations, times, traffic density, or enroll in driving lessons. A Fail rating indicated that the driver exhibited behavior that had a high probability of leading to crashes, but could not be easily remediated.

Naturalistic Driving

Assessment Low profile cameras (3.5 inches wide by 2.5 inches deep by 3 inches tall) were installed on the same day of passing the road test. Cameras were placed on the dash and in the back of the vehicle. One camera faced the participant, another faced forward, and the back two cameras faced diagonally forward to capture the environment to the sides of the vehicle. Participants were instructed to drive in their typical environment and to follow their daily routine. Video was downloaded to a digital video recording device that was placed under the passenger seat. Cameras were installed for two weeks. The research assistant reviewed all driving segments acquired and provided approximately four hours of consecutive daytime driving to the driving instructor for review. Only daytime driving was rated because nighttime video quality was suboptimal for coding.

The video ratings were completed one month after the road test by the same driving instructor to maximize comparability of ratings across environments. The instructor was not allowed to review his ratings from the road test. Naturalistic driving was rated with the Composite Driving Assessment Scale (CDAS), a measure developed for this study. Scale content was based on input from the study driving instructor’s standard assessment and from the Rhode Island Road Test Rating Scale. Behaviors were divided into discrete events (i.e., maneuvers) and global events (i.e., attention, attitude, reaction time). Because the number and types of driving situations in naturalistic driving assessment cannot be controlled in the same fashion as on a road test, each item was given a global rating of “unimpaired,” “mildly impaired,” or “moderately to severely impaired” according to the same 3-point Likert scale as the RIRT (range 0-2). Total scores ranged from 0 to 60 with higher scores reflecting poorer performance. The error severity score was an average score reflecting the sum of the ratings for each behavior divided by the maximum number of observed behaviors. The driving instructor also made a global rating of safety identical to the RIRT. Inter-rater agreement for the global ratings between two separate driving instructors rating 20 videos was in the moderate range for the global ratings (kappa =.45) and error scores (Spearman’s rho =.62, p<.05) 26.

Course difficulty for naturalistic driving was rated on a 10-point Likert scale ranging from 1 (simple) to 10 (extremely challenging). One rating was made for the four hours of driving to reflect the modal driving environment.

Analyses

For the participant who retested on the road test and remained in the study, the result of the initial road test was included in analyses. Group comparisons for baseline characteristics were made using independent-sample t-tests for continuous variables and χ² or Fisher’s exact test for categorical variables. A linear regression model was applied to examine the association between driving errors and history of crashes. Analyses were performed using S-Plus 8.2 ³⁴.

RESULTS

Demographic characteristics are presented in Table 1 for participants who passed the road test and had cameras installed in their cars (N=103). The final group consisted of 44 healthy participants, 41 patients with a CDR rating of.5 (questionable to very mild dementia) and 18 with a CDR rating of 1 (mild dementia). Given the small sample size of patients rated CDR 1, patients were combined into one group for all comparisons. Patients were older [t(101)=-3.44, P<0.001], less educated [t(101)=3.58, P<.001)], and drove fewer miles per week [t(101)=5.04, P<0.001] than the healthy participants. In terms of driving history, both groups reported very few crashes during the three years prior to enrollment (n=14), or history of violations in the year prior to enrollment. There were no differences between crash rate per year [t(101)=-1.08, P=0.28] or percent violations between the two groups, [Fisher test, P=0.17].

Table 1.

Demographic and driving characteristics of study sample.

	Healthy Controls (N=44)	Patients (N=59)	Total (N=103)
Age, y	71.2 (7.6)	76.0 (6.0)*	73.9 (7.2)
Gender (% Male)	38.6	49.2	45.6
Race (% Caucasian)	100.0	91.5%	95.2
Education, y	16.3 (3.8)	13.8 (3.4)*	14.9 (3.8)
MMSE	29.5 (0.7)	25.2 (2.8)*	27.0 (3.0)
Years Driving	53.1 (6.8)	55.5 (9.9)	54.5 (8.7)
Miles Driven per Week	200.8 (114.9)	98.8 (90.5)*	142.4 (133.1)
Crashes in Past Year (percent)	13.6	8.5	10.7
Crashes per Year/10,000 Miles Driven	0.2 (0.4)	1.4 (7.5)	0.8 (5.7)
History of Violations in Past Year (percent)	13.6	5.1	8.7

Values are mean (± SD) or %.

^*patients different from healthy controls at p<.001.

MMSE = Mini-Mental State Examination.

To address any potential biases in the patient group, refusers (n=64) were compared to enrolled patients. Compared to enrolled patients, refusers were similar in age, t(121) = 0.47, P = 0.64 (M for refusers=76.48, SD = 6.22) and gender (37% male refusers; デ²=2.44, P =0.12). Refusers were, however, more cognitively impaired as measured by the MMSE, t(121) = -2.95, P < 0.01 (M for refusers = 23.59, SD = 3.20).

Table 2 presents group comparisons of driving outcomes. Patients made more errors and were rated more poorly on global ratings of safety on the road test and naturalistic driving compared to cognitively healthy participants. The complexity of driving environments was comparable between groups, despite a reduction in the number of miles driven per week in the patient group.

Table 2.

Group differences on the road test (RIRT) and naturalistic driving (CDAS).

	Healthy Controls (n= 44)	Patients (n=59)	t	p-value
Road Test Error Score mean (± SD)	0.04 (0.03)	0.08 (0.06)	−4.20	<0.001
Global Safety Rating (RIRT, %)
Pass (no recommendations)	56.82	20.34
Pass (with recommendations)	29.55	32.20
Marginal (with restrictions)	11.36	30.51
Marginal (with training)	2.27	15.25
Fail	0.00	1.69		<0.001
Naturalistic Driving Error Score mean (± SD)	0.10 (0.08)	0.19 (0.13)	−4.08	<0.001
Global Safety Rating (CDAS; %)
Pass (no restrictions)	61.36	35.59
Pass (with recommendations)	22.73	20.34
Marginal (with restrictions)	13.64	20.34
Marginal (with training)	2.27	16.95
Fail	0.00	6.78		0.01
Naturalistic Driving Course Difficulty mean (± SD)	5.80 (1.15)	5.68 (1.32)	0.47	0.64

RIRT = Rhode Island Road Test; CDAS = Composite Driving Assessment Scale.

Error scores reflect the average score on a 0-2 Likert scale (0=normal; 1-mildly impaired; 2-moderately-severely impaired.

Course difficulty was rated on a 10-point likert scale with 1=simple and 10=extremely challenging.

In the overall group, ratings on the road test and naturalistic driving showed fair agreement, kappa=.33 ³², reflecting an effect size in the moderate range³⁵. The majority of participants passed both the road test and naturalistic driving (53%). Only four participants (4%) failed naturalistic driving after receiving a “pass with recommendations” (n=2), “marginal with restrictions” (n=1), or “marginal with training” (n=1) on the road test. Error scores in the two environments were correlated, r=0.41, P <.001, but error score severity was higher in the naturalistic environment (M=0.15, SD=0.12) compared to the road test (M=0.06, SD=0.05), t(102)=-8.31, P <.001.

Tables 3 and 4 show specific types of driving errors ranked by severity in both driving environments; only the ten maneuvers rated most poorly out of the entire CDAS and RIRT measures are presented for brevity. On average, specific maneuvers were rated as mildly impaired in both environments. Although the CDAS and RIRT do not rate identical maneuvers, inspection of the most severe error types across environments shows overlap among half of the maneuvers, including checking blind spots, making complete stops, lane keeping, traffic awareness, and speed control. Checking blind spots was the most notable error made in both environments. Between group comparisons for each of these maneuvers showed that the cognitively impaired participants were more impaired than healthy controls on a higher number of maneuvers on the road test (6/10 maneuvers) compared to naturalistic driving (2/10 maneuvers).

Table 3.

Road test (RIRT) errors ordered by severity in the overall group.

RIRT	Mean Rating
	Overall (N=103)	Controls (n=44)	Patients (n=59)
Checks blind spotsa	0.87	0.62	1.06*
Uses mirrors for lane change	0.25	0.13	0.34*
Uses mirrors	0.19	0.08	0.27*
Traffic awarenessa	0.19	0.10	0.26*
Speed controla	0.18	0.14	0.21
Approaches intersection at appropriate speed	0.17	0.19	0.16
Makes complete stopsa	0.17	0.12	0.21
Proceeds timely	0.17	0.09	0.23*
Brakes smoothly and accurately	0.14	0.13	0.15
Lane keepinga	0.14	0.07	0.20*

^*Note. denotes significantly poorer performance in the cognitively impaired group compared to the healthy participants; ratings were made on Likert scale ranging from 0-2 (0=intact; 1=mildly impaired; 2=moderately-severely impaired);

^aindicates overlap with naturalistic errors.

Table 4.

Naturalistic driving errors (CDAS) ordered by severity in the overall group.

CDAS items	Mean Rating
	Overall (n=103)	Controls (n=44)	Patients (n=59)
Checks blind spotsa	0.77	0.41	1.03*
Makes complete stopsa	0.64	0.60	0.67
Pays attention	0.51	0.48	0.54
Awareness of driving on others	0.51	0.32	0.66
Responds to signage	0.48	0.36	0.56
Lane keepinga	0.42	0.34	0.48
Appropriate response to emergency vehicles	0.38	0.25	0.47
Traffic awarenessa	0.38	0.25	0.47
Right turn	0.37	0.14	0.54*
Speed controla	0.36	0.32	0.39

^*Note. denotes significantly poorer performance in the cognitively impaired group compared to the healthy participants; ratings were made on Likert scale ranging from 0-2 (0=intact; 1=mildly impaired; 2=moderately-severely impaired.

^aindicates overlap with road test errors.

To determine if road test or naturalistic driving performance is a more sensitive measure of driving risk, history of crashes, corrected for miles driven per week during the three years prior to enrollment, was entered as a covariate in a linear regression model predicting error scores on the RIRT and CDAS. Two separate models were run using the RIRT and the CDAS as outcomes. Crash history was associated with error scores on the RIRT, F(1,101)=7.38, P =0.01, R²=0.07, but not on the CDAS, F(1,101)=0.60, P =0.44, R²=0.01.

Disease severity and age were then examined to determine if there were any demographic or disease characteristics that may affect the relationship between road test and naturalistic driving performance. As expected global cognitive status, as measured by the MMSE, was correlated with both the road test, r=-0.32, P <.01, and naturalistic driving, r=-0.22, P =0.02, with poorer performance on the MMSE associated with higher driving error scores in both settings. Age was correlated with the road test, r=0.26, p=0.01, but not naturalistic driving, r=0.10, P =0.30. When the groups were separated, for patients there was no relationship between age and performance on the road test, r=0.03, P =0.64, or naturalistic driving, r=0.13, P =0.32. For the healthy older adults, younger age was correlated with lower error severity on the road test, r=0.40, P =0.01, and greater error severity for naturalistic driving. r=-0.32, P =.04.

DISCUSSION

Individuals with cognitive impairment had higher errors scores and poorer global ratings of driving competence on both the road test and naturalistic driving compared to cognitively healthy older adults. Interestingly, the cognitively impaired participants had a greater number of more impaired maneuvers than controls on the road test than naturalistic driving, suggesting that the road test may be a more cognitively demanding task than naturalistic driving. These results are consistent with several studies demonstrating that mild cognitive impairment or early Alzheimer’s disease impacts driving skills, even if the person is able to pass a road test ^{6, 23}. Furthermore, global cognitive status (MMSE) was associated with driving errors in both settings. Taken together, findings suggest that both methods of driving assessment are sensitive to cognitive impairment beyond the effects of normal aging.

Given the limitations of the road test, it was of primary interest to determine if performance on a standardized road test related to individuals’ driving behavior in their own environment. Results showed fair agreement between global ratings of safety and error scores on the road test and naturalistic environment, suggesting that the road test is a reasonable proxy for driving behavior in the naturalistic setting for older adults with and without cognitive impairment. This is encouraging and suggests that the road test may not necessarily enhance driving performance through structure and cueing ¹⁷ or conversely reveal problems due to anxiety or course/vehicle unfamiliarity that would not be issues in the participants’ typical driving environment ¹⁵.

For both groups, more errors were committed in the naturalistic setting than on the road test, but this did not affect global ratings of safety. It is possible that the types of errors detected in the naturalistic setting were judged by the driving instructor to be less severe than the errors made on the road test. Alternatively, ratings may differ because the driving instructor had more opportunities to make judgments about errors compared to the much shorter driving test. Errors may have also been interpreted differently during the in-car experience compared to the purely observational ratings done by video recordings for naturalistic driving. Qualitatively, the types of errors made on the road test and naturalistic setting were similar with half of the error types overlapping between the two environments. Failure to check blind spots was the most egregious error in both settings, and awareness of traffic, making complete stops, speed control, and lane keeping were other error types frequently identified in both settings. These types of errors are generally consistent with other studies showing that older adults with cognitive impairment have significant difficulties with lane checking, lane changing, and merging, all of which require checking of blind spots ^{6, 7}.

It was of interest to determine if higher error scores in the naturalistic setting were related to a real-world safety measure. To accomplish this, history of crashes three years prior to enrollment was recorded and corrected for miles driven. This correction was made because past studies have shown that patients with cognitive impairment have higher crash rates when correcting for miles driven ^{23, 36}. In the current study, crashes per mile driven were positively associated with road test but not naturalistic driving errors, suggesting that the road test may be a more sensitive measure of crash risk. However, it is important to emphasize that in this group of participants, history of crashes was rare, and the majority had no crashes. Report of crash history was also self-report, so the reporting could have been biased. We also excluded participants with an at-fault crash within one year of enrollment, so we may have excluded the highest risk drivers. It is also possible that this relationship was biased by the few participants who did have crashes. These findings should be replicated in a sample with a greater range of crash history.

As expected, disease severity was related to both road test and naturalistic driving ratings. Interestingly, age had a unique effect on driving performance in each of these environments in the healthy older adults. More specifically, younger age was associated with fewer driving errors on the road test and greater driving errors in the naturalistic setting. This suggests that the younger participants were able to modify their behavior on the road test to “best behavior,” or that older participants modified their driving environment to reduce errors, but patients with cognitive impairment did not.

This study has limitations. Our evaluation of naturalistic driving was restricted by using video technology. We were unable to observe directional signal or pedal use, and we may have seen more overlap in error types between the two environments if these behaviors were measured in the naturalistic setting. The composition of our sample also limits the generalizability of these findings. Patients who refused to participate in the study were more cognitively impaired. As such, the sample was largely comprised of participants with cognitive impairment in the mild range, making it difficult to apply our findings to patients with more advanced dementia. In addition, our patients were slightly older and less educated than the healthy controls. Importantly, both groups were driving in comparably difficult settings during their naturalistic driving, despite patients driving fewer miles per week, suggesting that any differences in the complexity of naturalistic driving environments did not confound the findings. Finally, crash history and miles driven per week were self-reported, and could be biased.

Because of safety limitations, only drivers who passed the road test were allowed to be evaluated naturalistically. Consequently, we do not know how drivers deemed unsafe on the road test might have performed in naturalistic driving. It is interesting to note that the patient who initially failed the road test was judged to be safe based on their video recorded naturalistic driving. Conversely, four participants who passed the road test ultimately failed naturalistic driving. Overall, the positive predictive value of safe driving for the road test appears to be acceptable; however, the negative predictive value cannot be assessed since the worst drivers who failed the road test did not have an assessment of naturalistic driving. Of note, the instructor was allowed to provide advice on driving restrictions (e.g., restrict driving to local areas, avoid driving at night or on highways, drive with a co-pilot), which could have changed their behavior in the naturalistic setting. This factor, however, would have biased against finding a significant relationship between naturalistic and road test driving, suggesting that the relationship between the two settings may actually be more robust than the data indicate.

In conclusion, the road test appears to provide a reasonable estimate of driving in the naturalistic setting in older adults with and without cognitive impairment. Although our previous work suggests that different skills may be emphasized in the road test versus naturalistic environment ²⁶, those differences do not seem to impact the concordance between overall safety ratings and errors in the two environments. It is important to emphasize that these results do not adequately address whether performance in the naturalistic setting or during a road test is a better predictor of driving safety in this population. Future longitudinal naturalistic studies that include safety outcomes are needed to make this determination.