Interrater Reliability of the Record of Driving Errors (RODE)

Researchers examined the RODE's interrater reliability using 2 occupational therapy driving rehabilitation specialists working with 24 older adults diagnosed with dementia.

Abstract

The Record of Driving Errors (RODE) is a novel standardized tool designed to quantitatively document the specific types of driving errors that occur during a standardized performance-based road test. The purpose of this study was to determine interrater reliability between two occupational therapy driver rehabilitation specialists who quantitatively scored specific driving errors using the RODE in a sample of older adults diagnosed with dementia (n = 24). Intraclass correlation coefficients of major driving error and intervention categories indicated almost perfect agreement between raters. Using raters with adequate training and similar professional backgrounds, it is possible to have good interrater reliability using the RODE on a standardized road test.

Comprehensive driving assessments, usually performed by occupational therapy driver rehabilitation specialists (OT/DRSs), are used to determine whether a person is safe to continue driving when affected by a medical condition such as dementia. Typically, these assessments include a standardized clinical evaluation of vision and cognitive and motor skills and an on-the-road evaluation. The performance-based on-the-road test is the most widely accepted method for determining driving competency, with poor test results being associated with a history of at-fault crashes (Davis, Rockwood, Mitnitski, & Rockwood, 2011; De Raedt & Ponjaert-Kristoffersen, 2001; Fitten et al., 1995; Wood et al., 2009). In spite of being the most widely accepted measure of driving competency, road tests have limitations, which can include potential lack of objective scoring procedures, lack of established interrater reliability, variability in road courses and traffic conditions, and safety concerns (Lee, Cameron, & Lee, 2003; Stutts & Wilkins, 2003).

Road test performance has traditionally been classified using global scoring systems (e.g., safe, marginal, or unsafe). Although such global scoring systems lack reporting on specific driving errors, they have been found to have fairly good interrater reliability on overall driving performance (Brown et al., 2005; Carr, Barco, Wallendorf, Snellgrove, & Ott, 2011; Grace et al., 2005; Ott, Papandonatos, Davis, & Barco, 2012; Stav, Justiss, McCarthy, Mann, & Lanford, 2008).

Driving research is moving toward a closer examination of the types of specific errors that occur while driving. Driving errors have been classified in a variety of ways. Michon’s (1985) model is commonly used as a framework to examine driving errors operationally (operating the vehicle), tactically (maneuvering the vehicle), and strategically (navigating the vehicle). Some researchers have used various modifications of the Michon model to develop error scoring classifications. Other researchers have developed error scoring classifications that are based on state licensing scoring procedures (Dawson, Anderson, Uc, Dastrup, & Rizzo, 2009; Di Stefano & Macdonald, 2003) and used team consensus to distinguish more serious errors from less serious errors (Dawson et al., 2009). Researchers have also developed unique scoring classifications that include both the location where the driving error occurred (e.g., controlled intersections), the type of driving behavior error observed (e.g., lack of signaling), and the type of instructor intervention required (verbal or physical; Wood et al., 2009).

Studies have more recently attempted to objectify driving performance by using Likert-based scales to summarize driving characteristics or to quantify the number of errors possible per road test by recording errors at various points on a standardized road test (Akinwuntan et al., 2003; Anstey & Wood, 2011; Dawson et al., 2009; Fitten et al., 1995; Grace et al., 2005; Hunt et al., 1997; Janke & Eberhard, 1998; Odenheimer et al., 1994; Shechtman, Awadzi, Classen, Lanford, & Joo, 2010; Wood et al., 2009). These quantitative approaches may provide better understanding of the specific errors that occur with driving impairment, allowing OT/DRSs to better tailor driving recommendations to prolong or improve driving skills. However, interrater reliability information regarding road tests that addresses specific numbers and types of driving errors is lacking. Most interrater reliability studies do not include quantitative scoring methods, only global scoring methodology.

Studies on interrater reliability of quantitative scoring of driving errors (using a fixed-error or Likert scoring approach) are scarce in the literature. Moreover, the few published studies used varied approaches. One study compared scores from an in-car evaluator with those from a second evaluator reviewing a video of the same road test and had an intraclass correlation coefficient (ICC) of .62 for total errors (Akinwuntan et al., 2003). Another study reported interrater reliability on a quantitative error (fixed-error) scoring system and was able to reach a significant (but not high) correlation of .51 to .54 between two state driving examiners, one in the front passenger seat and one in the back seat (Janke & Eberhard, 1998). The purpose of the current study was to determine interrater reliability between two OT/DRSs who quantitatively scored specific driving errors using the Record of Driving Errors (RODE) in a sample of older adults diagnosed with dementia and referred for fitness-to-drive evaluations.

Method

Research Design

This cross-sectional study was part of a larger study investigating medical fitness to drive funded by the Missouri Department of Transportation (Carr et al., 2011). It was conducted at Washington University Medical School in St. Louis, Missouri, through the Driving Connections Clinic at The Rehabilitation Institute of St. Louis (TRISL) and approved by the Human Studies Committee at Washington University and the Health South Research Committee Corporate Office, a managing partner of TRISL.

Participants

Participants were recruited from January to August 2013 through the Memory Diagnostic Center of Washington University School of Medicine. Inclusion criteria required a diagnosis of dementia and a physician’s referral for a driving assessment, an active driver with current driver’s license, an Assessing Dementia–8 (AD8; Galvin, Roe, Xiong, & Morris, 2006) score of 2 or greater by the informant, a Clinical Dementia Rating (CDR; Hughes, Berg, Danziger, Coben, & Martin, 1982) of 1 or greater, at least 10 yr of driving experience, an informant available to answer questions and attend portions of the driving assessment, visual acuity acceptable in state driving guidelines, and ability to speak English. Exclusion criteria were any major chronic unstable diseases or conditions (e.g., seizures), severe orthopedic or musculoskeletal or neuromuscular impairments that would require adaptive equipment to drive, sensory or language impairments that would interfere with testing, use of sedating drugs, or a driving evaluation in the past 12 mo. All participants went through a comprehensive driving assessment that included the clinical assessment and on-road assessment (Carr et al., 2011).

Performance-Based Road Evaluation

The modified Washington University Road Test (mWURT; Carr et al., 2011) replaced the Washington University Road Test (WURT; Hunt et al., 1997) as a result of route changes caused by road construction. However, the mWURT uses common traffic situations from the WURT. The course currently consists of 14 right-hand turns (5 at stop signs, 3 at traffic lights, and 6 unprotected), 11 left-hand turns (6 at traffic lights, 5 unprotected), 33 traffic lights, and 10 stop signs. The road test begins in a closed, quiet parking lot and then progresses to low-traffic conditions (in a park setting) and on to moderate- to high-traffic conditions (e.g., complex intersections, greater traffic demands, traffic lights), as safety permits. The course is 13 miles long and takes approximately 1 hr to complete.

For this study, the road testing occurred in a midsized driving evaluation car with an occupational therapy certified driver rehabilitation specialist (OT/CDRS) positioned in the front passenger seat. The OT/CDRS provided directions and maintained safety of the vehicle. Two OT/DRSs used the RODE to record driving errors made by the participant. The two OT/DRSs sat in the back seat (one in the back middle and the other behind the passenger seat) with a large board between them to prevent them from viewing each other’s scoring tool. Both OT/DRSs who were scoring driving errors were blinded to results of any clinical testing.

Record of Driving Errors

The RODE, developed and implemented in 2007, is an objective approach to quantify the number and type of driving errors in a standardized performance-based road test. As noted previously, a paucity of published studies on scoring driving errors exists. The RODE incorporates knowledge of common driving errors that were identified by reading the literature and numerous case examples from clinically based driving evaluations. Different from most road test scoring approaches, the RODE allows for a continuous measure of errors throughout the road test, without a maximum limit. When driving, a variety of type and frequency of errors can occur at any given time between Point A and Point B. To capture the true extent of the occurrence of the errors, it is necessary to measure the errors not only at designated points along the route but also consistently throughout the whole driving experience. The RODE incorporates categories of driving errors (i.e., driving situation and driving behavior errors) from previous studies (Wood et al., 2009) and operationally defines specific errors within each category.

The RODE scoring methodology follows a three-tiered approach. First, all driving errors are recorded according to three levels of traffic condition in which they occur (closed parking lot, low traffic, and moderate to high traffic). Second, within each traffic condition, driving errors are recorded according to the specific driving situation that is involved (parking or backing, left turn, right turn, straightaway driving, or lane change). Third, within each driving situation, errors are further described by the type of driving behavior errors that occur (operational, tactical, or information-processing errors; Figure 1).

Figure 1.

Record of Driving Errors scoring.

Building on the Michon model, the RODE defines operational errors as those that involve the direct operation of the car (e.g., use of turn signals and foot pedals), tactical errors as those that require driving tactics or skill in traffic (e.g., maintaining lane position, visual scanning, yielding), and information-processing errors as those related to higher processing skills (e.g., following directions and using memory, making decisions, anticipating maneuvers, knowing traffic signs and rules of the road). Information-processing errors are analogous to but extend beyond the “strategic/planning” driving errors described in other studies (Grace et al., 2005; Withaar, Brouwer, & van Zomeren, 2000). The RODE uniquely acknowledges that multiple driving behavior errors can simultaneously occur and can result in one driving situation error. For example, a driving situation error in making a left turn could be the result of multiple driving behavior errors (such as failure to signal, failure to yield to a vehicle or pedestrian, and turning from an incorrect lane). Finally, the evaluator records any necessary interventions to maintain safety (e.g., verbal cue, wheel intervention, brake intervention) and whether the errors resulted in an occurrence of a dangerous event (e.g., errors that resulted in immediate, imminent danger of a collision or hazard; see Figure 1).

Training on Scoring

We implemented a rigorous training protocol for scoring driving errors. Training occurred over a 3-mo period, and the primary investigator (Barco, an OT/DRS with 15+ years of experience in driving evaluation) served as the trainer. Training included approximately 30 hr of written and verbal practice activities involving scoring methodology (familiarity with the scoring sheets, tallying of errors, and objective definitions of the errors and driving situations) and 15 hr of practice trials of recording errors during road testing to ensure understanding of the methodology, discuss and resolve differences in scoring, and demonstrate proficiency in the administration of the RODE. All training was completed before data collection.

Statistics

Interrater reliability was assessed using ICCs (Shrout & Fleiss, 1979). ICCs reflect the degree of agreement between raters. Higher ICCs indicate better reliability, with guidelines as follows: ≤0 = poor, .01–.20 = slight, .21–.40 = fair, .41–.60 = moderate, .61–.80 = substantial, and .81–1.00 = almost perfect agreement (Landis & Koch, 1977).

Results

Twenty-four participants met the inclusion criteria. Participants had CDR scores that reflected very mild (0.5, n = 20) or mild (1, n = 4) dementia. Their mean AD8 score was 5.7 (standard deviation [SD] = 1.8), and their mean score on the Short Blessed Test (Katzman et al., 1983), a brief test of memory, concentration, and orientation, was 7.0 (SD = 5.5). Seven (29.2%) participants were women, and 3 (12.5%) were of minority race. Participants were age 69.1 yr (SD = 9.3 yr) on average and had a mean of 15.3 yr of education (SD = 3.5 yr).

We calculated ICCs and associated 95% confidence intervals for situational (Table 1), driving behavior (Table 2), and intervention (Table 3) errors. Using the guidelines of Landis and Koch (1977), we determined that ICCs indicated almost perfect agreement between raters for total situational errors; total situational errors for closed lot, low traffic, and moderate to high traffic (see Table 1); total operational, tactical, and information-processing errors (see Table 2); and all intervention error categories (see Table 3). However, some of the situational and driving behavior subcategories that make up the totals showed lower interrater reliability (see Tables 1 and 2).

Table 1.

Reliability of Situational Errors

		95% CI
Situational errors	ICC	LL	UL
Total situational errors	.97	.93	.99
Total situational errors, closed lot	.84	.67	.93
Total situational errors, low traffic	.90	.79	.95
Right turn	.85	.69	.93
Straightaway	.44	.07	.71
Total situational errors, moderate to high traffic	.97	.93	.99
Right turn	.89	.77	.95
Left turn	.89	.77	.95
Straightaway	.96	.91	.98
Lane change	.67	.39	.84
Open parking lot	.88	.75	.95

Note. CI = confidence interval; ICC = intraclass correlation coefficient; LL = lower limit; UL = upper limit.

Table 2.

Reliability of Driving Behavior Errors

		95% CI
Behavior errors	ICC	LL	UL
Total operational errors	.91	.81	.96
Use of controls	.31	–.03	.65
Use of gears	.88	.75	.95
Turn signal use	.91	.81	.96
Pedal control	–.02	–.40	.37
Steering control	–.02	–.40	.37
Total tactical errors	.95	.89	.98
Visual scanning	.72	.47	.87
Distance judging	.35	–.04	.65
Lane use and position	.97	.93	.99
Stopping inappropriately	.5	.14	.74
Speed (slow)	.86	.71	.94
Speed (fast)	.77	.55	.89
Yield to vehicle	.37	–.02	.66
Yield to pedestrian	.66	.37	.83
Total information-processing errors	.84	.67	.93
Failure to anticipate	–.07	–.44	.32
Attention	.45	.08	.71
Decision making	.64	.34	.82
Memory (following directions)	.81	.62	.91
Lack of knowledge of rules of the road	.73	.48	.87
Speed of processing	.32	–.07	.63

Note. CI = confidence interval; ICC = intraclass correlation coefficient; LL = lower limit; UL = upper limit.

Table 3.

Reliability of Intervention Errors

		95% CI
Intervention errors	ICC	LL	UL
Verbal cueing	.98	.96	.99
Wheel interventions	.94	.87	.97
Brake interventions	.96	.91	.98
Dangerous actions	.93	.85	.97

Note. CI = confidence interval; ICC = intraclass correlation coefficient; LL = lower limit; UL = upper limit.

Discussion

Although quantitative error counts in driving research are becoming more common, interrater reliability information regarding these measures is scarce. For the RODE, we found strong interrater reliability across the important main categories of driving errors that occur during the on-road assessment. The stronger interrater reliability of this tool compared with interrater reliability of other quantitative error scoring is likely attributable to the intense training program in error observation using the RODE, similar professional background of both raters, standardized road course, and objective definitions of error types. Moreover, the raters were both positioned in the back seat compared with one scorer in the front seat and one in the back, as in Janke and Eberhard (1998). Seating position in the car has the potential to influence the type of driving errors that are observed and subsequently recorded because different seating positions give different views of the driver and the road.

Although the majority of driving behavior errors showed strong interrater reliability, some specific driving behavior errors had lower interrater reliability. The lower interrater reliability could be the result of a lower frequency of occurrence of the specific errors. The lowest ICCs tended to be found for errors that occurred with low frequency. For example, errors in pedal control (ICC = –.02) and in steering control (ICC = –.02) each occurred only one time for one participant (see Table 2). Moreover, lower rating on some specific errors could be the result of different OT/DRSs emphasizing different driving behaviors. Ott et al. (2012) found that different driving evaluators weighted various driving behaviors differently, which affected interrater reliability on scoring road test performance.

Limitations

This study had several limitations. Although the road course was standardized, the traffic occurrences on each route were not; when driving in real traffic conditions, it is not possible to standardize the various traffic-related situations that occur at any given time on the performance-based road course. Moreover, the study was limited to one site and a small sample. Replication with larger samples and other standardized routes needs to occur to increase generalizability of these results.

Recommendations for Future Research

Future studies need to address how the types and quantity of errors affect passing or failing a road test in a comprehensive driving assessment. Studies should examine how the types of errors may differ across different medical impairments and whether the RODE can be validated with prospective or retrospective crash history.

Because the RODE uses a continuous measure of errors across a performance-based road test (in contrast to a point system at predesignated locations), it may be more easily transferred to use at other locations. Future studies are necessary to test the reliability and validity of the RODE across various sites and with different performance-based road tests.

It is unknown how the seating position in the car affects evaluator observation of driving errors. Therefore, future research could focus on whether error observations using the RODE differ because of evaluator seating position (back seat vs. front seat). Studies should also focus on the comparisons between error observations that are direct from an evaluator and errors that are detected with instrumented vehicles (i.e., vehicles equipped with various electronic mechanisms and sensors to record and review driving performance later, using video analysis). Most clinically based occupational therapy road assessments include only one driving evaluator (positioned in the front seat) who is responsible for observing errors, providing directions, and maintaining safety of the vehicle. In such cases, it may not be possible for this person to record errors with the level of detail included in the RODE. Future studies will focus on ways to adapt the RODE to make it more usable in clinically based driving assessments.

Implications for Occupational Therapy Practice

The results of this study have the following implications for occupational therapy practice:

• With adequate training and objective definitions, it is possible for OT/DRSs to reliably identify driving errors on a standardized performance-based road assessment using the RODE.

• Being able to reliably identify the type of driving errors that occur during the road assessment is a beginning step toward identifying the type of occupational therapy–based interventions needed for improving and prolonging driving ability to maintain independence and participation throughout the life span.

• Care should be taken when using any complex driving error scoring tool, such as the RODE, to not distract the OT/DRS (in the front passenger seat) from effectively managing safety of the vehicle during the road assessment.