Prospective validation of a screening tool to identify older adults in need of a driving evaluation

Abstract

Background/Objectives

In at-risk older drivers, behind-the-wheel (BTW) tests can assess skills and inform vehicle modifications, retraining or, if necessary, the need for driving retirement. We sought to prospectively validate and refine the five-item “CRASH” screening tool for identifying older drivers needing a BTW test.

Design

Prospective observational study.

Setting

Geriatric and internal medicine primary care clinics affiliated with a tertiary care hospital, and a local BTW program.

Participants

315 cognitively-intact drivers aged ≥65 years.

Measurements

Participants completed: baseline questionnaire (including CRASH tool) and assessments; BTW test (evaluator blinded to questionnaire results); and one-month telephone follow-up. Analysis included descriptive statistics and examination of predictive ability of the CRASH tool to discriminate “normal” (pass) versus “abnormal” (conditional pass or fail) on the BTW test, with logistic regression and CART techniques for tool refinement.

Results

266 participants (84%) had a BTW test; of these, 17% had a normal and 83% an abnormal rating. Half (45%) of those with an abnormal score were advised to limit driving under particular conditions. Neither the CRASH tool nor its individual component variables were significantly associated with the summary BTW score; in refined models with other variables, the best-performing tool had approximately 67% sensitivity and specificity for an abnormal BTW score. Most participants found the BTW test useful and were willing to pay a median of $50. At one-month follow-up, no participants had stopped driving.

Conclusion

The CRASH screening tool cannot be recommended for use in clinical practice. Findings on older adults’ perceived utility of the BTW test and the stability of driving patterns at one-month follow-up could be useful for future research studies and for design of older driver programs.

INTRODUCTION

Driving helps older adults maintain independence, mobility, and community involvement,^{1, 2} but some drivers are at increased risk of crashes^{1, 3} because of medical conditions, medications, or age-related skill deterioration. Behind-the-wheel (BTW) driver evaluations are the criterion standard for assessing driving ability,⁴ although not every older driver requires or would benefit from such testing.

Primary care providers are uniquely positioned to initiate brief screening of older drivers as part of a tiered driver assessment program, with referral of screen-positive patients for BTW tests.⁵ A valid screening tool comprising a brief questions and a simple scoring system, similar to other decision rules used in clinical care, would allow clinicians to screen drivers efficiently and prioritize BTW testing resources for those at highest risk and those most likely to benefit.

Older driver screening tools already exist for self-administration,^6–8 and for cognitively-impaired drivers,^9–11 but many are impractical in busy clinical settings,^{e.g. 3, 6, 12–15} require family input,^{10, 15} or have not been prospectively validated with objective driving outcomes.^{10, 15, 16} Still missing is a valid, brief and simple method for clinicians to screen for driving risk in the general older adult population. In pilot work with a retrospective outcome, we developed the five-question CRASH tool¹⁷ by identifying questions associated with a recent crash. Here, we sought to prospectively validate and refine the CRASH tool in an independent sample against BTW test performance.

METHODS

Design, Setting & Participants

Older drivers were recruited from three primary care clinics affiliated with an urban academic hospital: a geriatric clinic, a general internal medicine (GIM) clinic located within the hospital, and an off-site GIM clinic. Eligible patients were ≥65 years of age, had driven within 30 days, spoke English, and were not significantly cognitively impaired by report or by a “Six-Item Screener” score ≤4.¹⁸ A random sample of potentially eligible patients (drawn from clinic rosters) received informational letters and up to three calls. Written informed consent was obtained and the Colorado Multiple Institutional Review Board approved this study.

At the baseline assessment, participants self-completed the CRASH screen and a questionnaire about their health and driving habits.⁸ Research staff administered physical, mental, and sensory function tests related to driving ability (visit time ≤60 minutes)^19–25 and saved data to secure online databases.¹²

Enrollees were referred to DriveSafe, an independent driving school where one-hour BTW tests were completed by one of two state-certified evaluators blinded to participants’ baseline data. Evaluators used a standardized BTW assessment protocol with a standard route adaptable to the individual driver’s performance.²⁶ The session evaluated critical driving skills: vehicle position; lane maintenance; speed regulation; yielding; signaling; visual scanning; adjustment to stimuli; and gap acceptance.²⁷ Either the evaluator or driver could terminate the session at any time, but this did not occur with any study sessions. For inter-rater reliability, both evaluators simultaneously but independently assessed a sample of participants, alternating from either the front or back seat.²⁷

Participants were asked to complete the BTW session within one month of enrollment and received a $25 gift card after BTW completion. The BTW session, plus one later retraining session (if recommended), was free for the participant. Research staff contacted participants approximately one month after enrollment for a telephone survey.

Variables

The primary outcome was the summary rating from the BTW test: pass (safe); conditional pass (safe with restrictions or recommendations, or unsafe but remediable); or fail (unsafe and not remediable). For this study, we decided a priori to dichotomize scores as either normal (safe) or abnormal (conditional pass or fail) to identify those individuals most in need of referral for a BTW test.

Baseline self-reported variables included: demographic characteristics; personal health assessment; need for assistance with activities of daily living (ADLs); and driving experiences and attitudes. Baseline functional assessments included: Rapid Pace Walk;^{19, 20} Montreal Cognitive Assessment (MoCA);²¹ Timed Up and Go;²² visual fields by confrontation;²³ 4-meter walk gait speed test;²⁴ Gross Impairments Screening (GRIMPS) arm reach and head/neck rotation tests;²⁰ and one leg (uni-pedal) stance test.²⁵ All tests were administered and scored by staff according to published norms (age-adjusted when indicated by scoring instructions).

Data Analysis

We described characteristics using proportions and 95% confidence intervals (CI) or medians and interquartile ranges (IQR). We assessed differences between groups with normal or abnormal BTW scores using chi-square, Fisher’s Exact, or Wilcoxon tests, as appropriate. Using established methods for developing and validating a risk score, we examined receiver operating characteristics (ROC) curves and tool sensitivities and specificities for predicting the BTW score. In our derivation study, the CRASH tool had an area under the ROC (AUC) of 0.71.

For tool refinement, we used unadjusted and adjusted odds ratios to identify variables associated with a normal BTW test, using p<0.25 in bivariate analyses as initial criteria for multivariable model inclusion. We excluded and reintroduced variables to determine their effect on other variables and the final model, finally developing a model to maximize simplicity while attempting to maintain calibration and discrimination and minimize over-fitting. To augment this modeling approach, we used Classification and Regression Trees (CART) to develop a model predictive of BTW score. For CART analysis, we similarly included all variables as candidate predictors. Finally, we compared the three models (original CRASH tool, logistic regression model, and CART model) using ROC characteristics.

RESULTS

Of the 2,081 clinic patients reached by phone, we screened 881 for eligibility (Figure 1). Of these, 488 were eligible for participation and 315 enrolled (65% participation). Among participants, 266 (84%) completed the BTW protocol; those who did not schedule a BTW test were more likely to be unmarried and currently employed. Those who did not have a BTW test were included in analyses of one-month follow-up outcomes but not of the CRASH tool performance.

Figure 1. Study flow.

^aSix Item Screener ≤4

Among the 266 BTW test completers, 54% were male, 89% were white, and the median age was 73 (IQR 68-78; Table 1) years. Most were married, lived with at least one person, and had at least a college education; only 14% were currently employed. Most rated their health as excellent or very good, and only a minority reported a fall, hospitalization, or injury in the prior year. Although most participants rated their personal health and functioning highly in the baseline assessments, many had abnormal baseline scores on the Timed Up and Go (51%), MoCa (<26 seconds; 43%), GRIMPS Head and Neck (37%), and Rapid Pace Walk (>7 seconds; 33%).

Table 1.

Participant Characteristics, by BTW Score (n=266)

	Total		Normal score n=45		Abnormal score n=221		P
	n	%	n	%	n	%
Age group (years)							0.03
65-74 years	160	60	35	78	125	57
75-79 years	53	20	5	11	48	22
≥80 years	53	20	5	11	48	22
Male	143	54	32	71	111	50	0.01
Race
White	236	89	38	84	198	90	0.25a
Black or African American	22	8	4	9	18	8
Other	8	3	3	7	5	2
Non-Hispanic	251	97	43	96	208	97	0.66a
Currently married	163	61	30	67	133	60	0.42
Live in private home/apartment	251	94	45	100	206	93	0.08a
Live alone	73	27	11	24	62	28	0.62
College graduate or higher	177	67	30	67	147	67	0.95
Currently employed	38	14	11	24	27	12	0.03
Health self-rating							0.04
Excellent	48	18	14	31	34	15
Very good	102	38	14	31	88	40
Good/Fair/Poor	116	44	17	38	99	45
≥1 fall (past 12 months)	70	27	12	27	58	26	0.98
≥1 hospitalization (past 12 months)	51	19	12	27	39	18	0.17
≥1 injury (past 12 months)	55	21	7	16	48	22	0.35
Without help, have difficulty:
Walking 3 blocks without stopping	56	21	8	18	48	22	0.57
Walking up 10 steps without stopping	39	15	4	9	35	16	0.26
Carrying 10 lbs of groceries	38	14	3	7	35	16	0.12
Functional Assessments
Abnormal Rapid Pace Walk (> 7 seconds)19, 20	88	33	10	22	78	35	0.09
Abnormal MoCA Score (< 26)21	115	43	15	33	100	45	0.14
Abnormal Timed Up and Go (age-based scoring)22	136	51	21	47	115	52	0.51
Abnormal Visual Fields by Confrontation23	115	43	18	40	97	44	0.63
Mean/Median 4-meter walk gait speed test24	4.2	4	3.8	4	4.3	4.0	0.004b
Abnormal GRIMPS Arm Chair20	3	1	0	0	3	1	0.99a
Abnormal GRIMPS Head and Neck20	98	37	15	33	83	38	0.58
Mean/Median one-leg stance25	14.4	12	17.9	19	13.7	10	0.02b

^aFisher’s Exact test

^bWilcoxon test

BTW: Behind the Wheel

When asked about their driving experiences and attitudes, most rated their driving as better than others’ and few reported a police stop (12%) or car crash (11%) in the prior year (Tables 2 & S1). Overall, most expressed positive attitudes towards driving (including enjoyment and confidence levels), though some reported frustration, stress, anxiety or criticism from others. Some reported difficulty with specific driving tasks, such as judging distances for parking (13%) or turning their head to back up or check for traffic (17%). While 32% acknowledged feeling their reactions were slower than they used to be, 84% said they felt their reactions were quick enough to handle a dangerous situation. Approximately a quarter reported difficulty with glare from headlights at night, difficulty reading unfamiliar signs, or being surprised by vehicles during merging. Almost two-thirds avoided driving in at least one circumstance, most commonly in snow or ice (38%), at night (32%), in heavy traffic (32%), in someone else’s car (18%), or for long distances (15%).

Table 2.

Driving Characteristics, by BTW Score (n=266)

	Total		Normal score n=45		Abnormal BTW score n=221		P
	n	%	n	%	n	%
CRASH questions
C (confusion): Ever feel confused, nervous, or uncomfortable while driving	28	11	5	11	23	10	0.80a
R (regular): Currently drive daily/almost daily	242	91	41	93	201	91	0.78a
A (avoid alone): Avoid driving when alone	3	1	0	0	3	1	0.99a
S (sight): Ever have trouble reading license plate on the car in front while stopped at traffic light	6	2	1	2	5	2	0.99a
H (hand over): Anyone recommended you stop driving or give up your car keys	4	2	0	0	4	2	0.99a
Experiences and attitudes
≥1 police stop (last 12 months)	32	12	10	22	22	10	0.02
≥1 motor vehicle crash (last 12 months)	28	11	4	9	24	11	0.80a
≥1 near miss crash while driving	42	16	7	16	35	16	0.93
≥1 minor fender bender with stationary cars/objects	27	10	6	13	21	10	0.43a
Enjoy driving	214	81	36	82	178	81	0.89
Feel confident in driving ability	233	89	38	86	195	89	0.61
Overall find driving stressful	23	9	2	4	21	10	0.39a
Often get frustrated or angry at other drivers	65	25	8	18	57	26	0.25
Do not avoid driving in any situations	83	31	17	38	66	30	0.30
Others criticize driving	47	18	7	16	40	18	0.67
Others refuse to drive with you (or prevent you from driving with certain passengers)	3	1	0	0	3	1	0.99a
Ever forgotten where you were going while driving	31	12	7	16	24	11	0.38
Dozed or nodded off for a moment while driving	16	6	3	7	13	6	0.74a
Prefer to drive with co-pilot	13	5	3	7	10	5	0.47a
When driving at night, bothered by the properly dimmed headlights of oncoming cars	58	24	11	26	47	24	0.72
Ever have any limitations in peripheral vision	12	5	1	2	11	5	0.70a
Have difficulty:
Getting into and out of a car	33	13	3	7	30	14	0.22
Holding the steering wheel firmly	21	8	2	5	19	9	0.54a
Turning head to back up or check for traffic	45	17	4	9	41	19	0.12
Pressing the brake pedal	21	8	2	5	19	9	0.54a
Moving between pedals quickly	19	7	1	2	18	8	0.33a
Pulling out into a busy street or freeway	23	9	1	2	22	10	0.14a
Backing up	30	11	2	5	28	13	0.11
Judging speed when approaching stopped vehicle	21	8	1	2	20	9	0.22a
Judging distances for parking	34	13	3	7	31	14	0.18

^aFisher’s Exact test BTW: Behind the Wheel

For the CRASH tool, the prevalence of positive responses was: Ever feel Confused or disoriented while driving? (11%); Regular [daily or almost daily] driver? (91%); Avoid driving alone? (1%); Ever have difficulty Seeing the license plate in front while stopped? (2%); Recommendation from someone to Hand over the keys in the past year? (2%). The distribution of scores (with one point given for a positive response to any of the five questions) was weighted towards low scores; 7% had zero, 80% one, and 12% two points, with only two participants having three points and none having four or five.

BTW Test Results

The median time from enrollment to BTW test was 21 days (interquartile range: 12–30; range: 1–84). Nineteen participants had two evaluators in the car for their BTW evaluation; agreement between these raters was 95% for the primary outcome (Kappa=0.83). For the dichotomous summary rating of the BTW test, 17% of participants had a normal rating (“pass”), while 83% had an abnormal rating (74% “conditional pass” and 9% “fail”). Evaluators recommended periodic re-evaluations for 13% of participants and retraining for 4.6%, along with a range of skills needing improvement (Figure S1). Even those with a “pass” score often had such deficiencies identified, though they may have been minor infractions (such as not coming to a complete stop at a stop sign). However, certain skills needing improvement were more often identified in drivers with an abnormal summary score; these included legal stops, turns, maintaining appropriate distance in traffic, and speed (Figure S1). Almost half (45%) of those with an abnormal score and 20% of those with a normal score were advised to limit driving in particular conditions. Among those with such a recommendation, the most commonly cited conditions were at night, in adverse weather or heavy traffic, on the freeway, or in unfamiliar areas (Figure S1).

Predictors of BTW Performance

The original CRASH tool poorly predicted BTW performance, with a ROC AUC of 0.51 (Figure S2). In univariate and multivariable logistic regression, none of the original five CRASH variables were significantly associated with the BTW summary rating either individually or in combination. Rather, the final multivariable logistic regression model associated with normal BTW performance included the following variables: age 75 to 79 (OR 0.4, 95%CI 0.1-1.0) or age ≥80 (OR 0.4, 95%CI 0.1-1.1) versus age 65-74 years; male gender (OR 2.5, 95%CI 1.2-5.2); and overall health self-rating of excellent (OR 2.0, 95%CI 0.9-4.7) or very good (OR 0.9, 95%CI 0.4-1.9) versus good, fair or poor health. The logistic regression model had a ROC AUC of 0.68 (Figure S2). The final CART model included age, gender, health self-rating, falls in the past 12 months, and highest grade completed in school, with a final ROC AUC of 0.67 (Figure S2).

Telephone follow-up

Among the 284 participants reached for telephone follow-up at 1 month, 18 (6.3%) said their overall health had changed but none had stopped driving. When asked about events in the past month, 8.1% reported at least 1 fall, 6.3% reported an injury requiring limited activities or medical attention, and 2.1% had been admitted to a hospital. About 3% reported either a motor vehicle crash (1.8%) or a police stop (1.1%). Fifty participants (18%) had talked with someone in the past month about plans to stop driving, more frequently a spouse/partner (32%), other family member (52%) or friend (18%) than a healthcare provider (4%). Among the 247 participants who completed both the BTW test and telephone follow-up, 68% said it was likely or very likely that the DriveSafe evaluation would change the amount or way they drove. Most gave high ratings to utility of the session and recommendations (data not shown). When asked how much they would be willing to spend for a similar BTW evaluation if they had to pay for it themselves, the median response was $50 (IQR: $20–$75; range: $0–$550), whereas the retail cost of evaluation was $99 at the time of this study.

DISCUSSION

In this prospective study of older drivers, we aimed to validate a five-question screening tool to identify which drivers should be referred for specialist testing. An ideal screening tool for a busy clinical setting would not require the time or expense of specific tests and would be independent of the physical examination. Additional ideal elements include not requiring family presence and a focus on functional deficits rather than the underlying etiology of those deficits, which would increase the tool’s face validity and ease of use.

Unfortunately, even with advanced modeling techniques, we were unable to validate the original CRASH tool¹⁷ or develop a modified version with adequate sensitivity or specificity for clinical use. For our primary outcome, we used objective performance on a standardized BTW road test, and our interrater reliability was excellent. Yet even these tests are not perfect, as an individual driver’s performance may vary day-to-day, and we could not control for weather or road conditions. In addition, most drivers had a “conditional pass” score on the test, while very few failed the test. For predictive variables, we used a range of self-reported factors along with brief physical and cognitive tests, all chosen because of their prior use in older driver evaluations or research. It is possible that a driver’s responses to these questions or tests might have changed in the time period between baseline and their BTW test, although at telephone follow-up very few reported major changes in health or driving patterns.

BTW tests are most typically administered by driving instructors or occupational therapists. Unfortunately, availability of programs is limited in the United States.²⁸ In our study, two thirds of participants said the BTW session would likely change their future driving, and overall program ratings were high. But cost remains a challenge: while the typical cost is approximately $400 for comprehensive evaluations (clinical evaluation plus BTW)²⁸ and $88 for BTW tests at driving schools,²⁹ our participants on average said they would be willing to spend only $50.

During the month after enrollment, no participants ceased driving. Low driving cessation rates in the short term may have implications for future studies that consider driving behaviors in older drivers through shorter time spans. Future studies should also examine changes in driving habits along with driving cessation, as research shows that older drivers are likely to modify their driving behavior first before completely ceasing driving.³⁰ Further follow up prospectively over 12-24 months could identify high risk groups amenable to intervention to prolong safe and active driving.

Study limitations include recall or reporting bias related to older driver self-report of driving behaviors and other screening tool questions. Screening questions might perform differently in other settings or populations; this sample from three outpatient sites included generally healthy, community-dwelling older adults without significant cognitive impairment or acute illness. In addition, although we assured participants that their responses were confidential, some may have declined participation because of fear of license revocation. Another issue relates to the power of our study; the final distribution of BTW scores limited our ability to consider multiple variables in the model. The summary score is the current gold standard in clinical practice, but in future work we hope to explore gradations within the scoring.

CONCLUSION

The overall goal of this line of research remains to develop an effective, efficient and fair approach to older driver testing. Unfortunately, in this study, no questions or assessments were able to predict BTW outcomes with sufficient sensitivity or specificity to recommend use in clinical practice. But these efforts should continue, especially as the population ages. Evidence-based tiered older driver assessment could give clinicians, drivers and family members a usable method for deciding when to “hand over the keys” without unnecessarily restricting older adults’ mobility.

Supplementary Material

Supp FigS1

Figure S1. Driving skills identified as needing improvement and recommendations of driving conditions to avoid, by BTW summary score (n=266)

*P<0.05 **P<0.01 ***P<0.001.

Supp FigS2

Figure S2. Receiver operating characteristics curve for models of factors associated with BTW performance (n=266).

CRASH model area under the curve (AUC)=0.51. Logistic regression model AUC=0.68. CART model AUC=0.67.

Supp TableS1

Table S1. Full Set of Driving Characteristics, by BTW Score (n=266).

^aFisher’s Exact test BTW: Behind the Wheel.

Italicized questions are those in the original CRASH tool.

Impact Statement.

We certify that this work is novel and also confirmatory of recent clinical research (including Betz et al; J Am Geriatr Soc 2012;60: 1791-1794). The potential impact of this research on clinical care or health policy includes further directing the development and implementation of tiered screening and assessment protocols for older drivers.