Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Mar 1.
Published in final edited form as: Accid Anal Prev. 2021 Jan 28;152:105986. doi: 10.1016/j.aap.2021.105986

Use of Multilevel Modeling to Examine Variability of Distracted Driving Behavior in Naturalistic Driving Studies

Sara A Freed 1, Lesley A Ross 2, Alyssa A Gamaldo 3, Despina Stavrinos 4
PMCID: PMC8204745  NIHMSID: NIHMS1668029  PMID: 33517207

Abstract

Current methods of analyzing data from naturalistic driving studies provide important insights into real-world safety-related driving behaviors, but are limited in the depth of information they currently offer. Driving measures are frequently collapsed to summary levels across the study period, excluding more fine-grained differences such as changes that occur from trip to trip. By retaining trip-specific data, it is possible to quantify how much a driver differs from trip to trip (within-person variability) in addition to how he or she differs from other drivers (between-person variability). To the authors’ knowledge, the current study is the first to use multilevel modeling to quantify variability in distracted driving behavior in a naturalistic dataset of older drivers. The current study demonstrates the utility of examining within-person variability in a naturalistic driving dataset of 68 older drivers across two weeks. First, multilevel models were conducted for three distracted driving behaviors to distinguish within-person variability from between-person variability in these behaviors. A high percentage of variation in distracted driving behaviors was attributable to within-person differences, indicating that drivers’ behaviors varied more across their own driving trips than from other drivers (ICCs= .93). Then, to demonstrate the utility of personal characteristics in predicting daily driving behavior, a hypothetical model is presented using simulated daily sleep duration from the previous night to predict distracted driving behavior the following day. The current study demonstrates substantial variability in driving behaviors within an older adult sample and the promise of individual characteristics to provide better prediction of driving behaviors relevant to safety, which can be applied in investigations of current naturalistic driving datasets and in designing future studies.

Keywords: multilevel modeling, variability, naturalistic driving, driver distraction, older drivers

1. Introduction

The proliferation of naturalistic driving studies in the last decade has presented an unparalleled opportunity for a greater understanding of driving behavior. Naturalistic studies, such as the 100-Car Naturalistic Driving study (Klauer, Guo, Sudweeks, & Dingus, 2010), LongROAD study (Li et al., 2017), and the SHRP2 study (Blatt et al., 2015), have made significant strides in capturing older adult driving behavior in the real world through the use of global position systems (GPS), video cameras, accelerometers, and other in-vehicle technologies. However, while many studies examine summary data, there is much more that can be gained from the large amount of data produced in these studies. Specifically, naturalistic driving studies provide valuable information on variability in driving behavior. In turn, driver characteristics that fluctuate day-to-day can be used to predict variability in driving behavior. The current study examines the utility of multilevel modeling in elucidating variability and predictors of variability in distracted driving behaviors in a naturalistic driving dataset.

Naturalistic driving studies record driving behavior over the course of several days through years, producing a large amount of data with a nested, or multilevel, data structure (i.e., multiple driving trips within each driver). A variety of reductionistic analytic approaches have been used with the data collected in naturalistic driving studies in order to address the nested data structure. For example, a study using two-week baseline driving data from the LongROAD study reported the percent of driving trips per month taken by older adults in certain situations, such as the percent of trips taken at nighttime, in rush hour, and on high-speed roads (Molnar, Eby, Bogard, LeBlanc, & Zakrajsek, 2018). Another study reported average duration of distracted driving episodes, mentioning but not statistically accounting for the fact that a driver may have contributed to multiple distracted driving episodes (Schneidereit, Petzoldt, Keinath, & Krems, 2017). Other studies extract events during the study period in which a crash or near-crash occurred (e.g., Huisingh et al., 2017; Klauer et al., 2013), limiting analyses to only a small number of events that may not be representative of typical driving behavior.

Analytic approaches to naturalistic driving data which remove or ignore the nested data structure inherent in these datasets are missing vital information on the ways in which driving behavior not only varies between drivers, but also varies from trip to trip within a driver. For example, a common way to analyze naturalistic data on speeding behavior is to calculate the proportion of all driving time during the study period in which a driver was driving above the speed limit. This method of condensing data prioritizes between-person differences (i.e., how much drivers differ from other drivers in their speeding behavior), while ignoring within-person differences (i.e., how much drivers differ from trip to trip in their speeding behavior). Research has begun to recognize the high variability in driving behavior but has not yet attempted to quantify this variability in a meaningful way. For example, one paper examining distracted driving in the SHRP 2 naturalistic dataset notes that drivers varied from one another in the effects of distracted driving duration on speed, though no statistical tests of this variability were conducted (Schneidereit et al., 2017). Quantifying this between-person variability and comparing it to how drivers varied from trip to trip in their own distracted driving duration and speed adjustments can provide valuable information on how driving safety and its antecedents fluctuate from trip to trip.

Multilevel modeling is an effective way of capturing and comparing within-person and between-person variability (Singer & Willett, 2003). Multilevel modeling accounts for nested data structure, such as multiple trips per individual within multiple individuals. Multilevel modeling produces statistics comparing within-person and between person variability and quantifying how much variability in a certain behavior is due to within-person versus between-person differences. Some naturalistic driving studies have used multilevel modeling to account for the nested data structure inherent in naturalistic driving studies but have not characterized the variability of driving behavior as an important descriptive and outcome variable. For example, an analysis of the 100-Car study used mixed-effects logistic regression, a type of multilevel modeling, to examine associations between distracted driving and crashes or near-crashes (Klauer et al., 2013). Another study used a multilevel modeling approach to examine how temporal and road characteristics of a driving trip are related to driver behaviors during a trip, such as speeding and braking (Ellison, Greaves, & Bliemer, 2015). This study examined two static driver characteristics, age and gender, as predictors of driving behavior. However, neither study reported the variability in driving behaviors within drivers as a descriptive characteristic of the sample.

Multilevel modeling is an important technique to understand safety-related driving behavior. Current uses of multilevel modeling in naturalistic driving studies are an important step to greater understanding of safety-related driving behavior. However, the capabilities of multilevel modeling for driving research extend beyond simply accounting for a nested data structure. Multilevel modeling can also provide in-depth information on variability in safety-related driving behavior and can answer the question: How much does a driver differ from another driver in his or her driving behaviors, and how much does an individual driver vary in his or her own driving behavior from trip to trip?

In addition to driving behaviors, driver characteristics are also likely to fluctuate from trip to trip but have not yet been explored as predictors of driving behavior in naturalistic driving studies. Sleep quality is a driver characteristic that is well-established as a between-person predictor of driving. For example, older adults who report difficulties falling asleep reported lower driving mileage and lower self-ratings of driving capacity than older adults who did not report difficulties falling asleep (Fragoso, Araujo, Van Ness, & Marottoli, 2008). Similarly, middle-age and older adults who get six hours of sleep are more likely to have a self-reported motor vehicle crash than adults who get seven to eight hours of sleep a night (Gottlieb, Ellenbogen, Bianchi, & Czeisler, 2018). Previous studies of sleep and driving have examined differences in sleep between drivers as a predictor of driving. However, sleep can fluctuate within a driver from day to day and may relate to driving safety as a within-driver association.

Indeed, psychological research has found that a number of driver characteristics related to driving fluctuate on a daily basis. For example, Gamaldo and colleagues examined variations in sleep duration across 8 days (Gamaldo, Allaire, & Whitfield, 2010). They found that about half of the variance in sleep duration was within-person, indicating that participants varied almost the same amount from themselves as they varied from others in sleep duration. Importantly, these results suggest that just knowing the sample average in sleep duration does not give an accurate picture of what sleep duration actually looks like from day to day for each person. They also found within-person associations between sleep duration and cognitive performance: when a person varies from his or her own average hours of sleep, he or she performs worse on cognitive tests the next day. This is especially important given the strong connection between fluid cognitive abilities and driving safety in older adults (Aksan, Anderson, Dawson, Uc, & Rizzo, 2015; Cross et al., 2009; Friedman, McGwin Jr, Ball, & Owsley, 2013; Huisingh et al., 2017). No studies to date have examined how within-person differences in sleep are related to within-person differences in driving behavior. Instead of examining how people who get an average of six hours of sleep per night differ from people who get seven to eight hours of sleep per night in crash rates, it would be informative to examine how people’s driving safety is different on a day when they got six hours of sleep the night before compared with a day they get seven to eight hours of sleep.

Distracted driving is an important behavior associated with driving safety that can be better understood by examination of within-person differences. Distracted driving is a major public safety issue for drivers across the lifespan: 18% of distracted driving-related fatal crashes involved older adults (60 years and older) in 2016 (National Highway Traffic Safety Administration, 2018). Related, 42% of adults between 60 and 74 years of age reported talking on a handheld cell phone while driving in the past 30 days, and 23% reported having read a text message or email on a cell phone while driving (AAA Foundation for Traffic Safety, 2019). There is a large body of evidence on the impact of secondary tasks on driving performance in both simulator and naturalistic work (see review by Ferdinand & Menachemi, 2014), and experimental studies suggest that distracted driving may be particularly detrimental to older adult driving safety (Aksan et al., 2013; Choudhary & Velaga, 2017; Fofanova & Vollrath, 2011; Thompson et al., 2012). However, prior surveys rely on self-report and do not provide information on variability, or whether drivers engage in these behaviors every time they drive or only some of the time. Additionally, no information is available on how daily sleep duration is related to daily driving behavior. Drivers may choose to engage or not engage in distracted driving behaviors when they get less sleep than their average the night before a trip. Understanding the nature of distracted driving behavior is key to understanding ways to prevent it and therefore improve driving safety, making it an excellent candidate for examining variability in distracted driving and predictors of this variability.

To our knowledge, no studies have used multilevel modeling to quantify variability in trip-level driving behavior in a naturalistic setting or the potential role of daily personal characteristics such as sleep in influencing driving behavior, making the present study the first of its kind. The current study demonstrates the use of a multilevel modeling approach to examine within-person and between-person distracted driving behaviors in older adults and examines the potential role of daily sleep as a predictor of distracted driving behaviors. The first aim was to characterize distracted driving behaviors of older adults during driving trips using objective assessments of real-world driving obtained in a naturalistic driving study. We hypothesized that a large amount of variability in distracted driving behavior across trips will be accounted for by within-person differences. The second aim was to demonstrate the potential of a personal characteristic, sleep duration, as a predictor of driving behaviors using simulated daily sleep data. As the sleep data are simulated based on a normal distribution, we do not expect any associations between simulated sleep and actual driving behavior, but aim to demonstrate the method of using within-person predictors of driving and the value of parameters obtained from this method.

2. Material and Methods

2.1. Study Design and Procedures

Secondary data analyses were conducted using the Senior and Adolescent Naturalistic Driving Study (SANDS). Participants were recruited through recruitment databases and community advertisements in a major city in the southeast United States. Inclusion criteria were: having a valid driver’s license and liability insurance, being the primary driver of personal vehicle, driving at least three days per week, being ages 65 and older, and demonstrating no evidence of dementia as assessed by a score of 22 or higher on the Modified Telephone Interview for Cognitive Status (TICS-M, de Jager, Budge, & Clarke, 2003). Participants’ vehicles were equipped at a baseline appointment with a Naturalistic Data Acquisition Device (N-DAD) and were instructed to drive as they normally would for a two-week period. After the two-week driving period, participants returned for a post-test appointment and removal of the N-DAD. Data collection occurred between November 2013 and July 2014. Participants 65 years of age and older with N-DAD data were included in the current study (n = 68). All study activities were approved by the University IRB and all participants signed an informed consent (see Stavrinos, Ross, & Sisiopiku, 2014 for further details).

2.2. Naturalistic Data Acquisition Device (N-DAD)

The N-DAD was developed to gather objective measures of real-world driving behavior. The N-DAD was installed in participants’ vehicles and mounted on the front windshield. The system consisted of an Android smartphone (Samsung HTC EVO 4G LTE®) with two wide-angle lenses. Three types of systems within the smartphone collected data: (1) An accelerometer collected data on acceleration (i.e., g-force events), (2) A global position system (GPS) collected latitude and longitude coordinates of the vehicle, and (3) A dual camera system collected photographic images approximately every two seconds of the front exterior of the vehicle and the interior of the vehicle. A 15-minute driving trip would have approximately 450 interior images or frames. Data collection by the N-DAD was initiated by vehicle motion and continued until no vehicle motion was detected for five consecutive minutes. A research assistant screened all videos and removed images not relevant to each drive (e.g., frames prior to the vehicle moving at the beginning of the drive and frames after the drive had ended). The N-DAD collected data on driving trips taken by the participant during the two-week study period. Photographic images were then coded for specific driving behaviors with acceptable inter-rater reliability at r = 0.90 (Stavrinos, Ross, & Sisiopiku, 2014). The N-DAD was tested and validated against a self-report driving log prior to the start of the SANDS project, with a significant relationship between self-reported and N-DAD collected trip time (r = 0.82, p < .001) and between self-reported and N-DAD collected trip distance (r = 0.94, p < .001).

2.3. Measures

2.3.1. Distracted Driving Behavior

Distracted driving behavior was assessed via the coding of the photographic images, or frames, collected by the N-DAD during driving trips taken by participants across the two-week study period. In some cases of low photographic data quality that limited view of the driver, such as too much light or not enough light, or a camera angle pointed too high or too low, coders were unable to determine if a behavior was present. For the images in which it could not be determined whether a behavior was present, the behavior was coded as missing. Eating, drinking, reading, grooming, interacting with a cell phone, smoking, and reaching were all coded as distracted driving behaviors. Table 1 describes how each distracted driving behavior was conceptualized in the coding process. The state where the study was conducted passed a texting ban with primary enforcement the year prior to data collection, though talking on a handheld cell phone was still legal.

Table 1.

Distracted driving behavior coding

Behavior Operationalization
Reaching Participant’s arm was extended to the radio, into the passenger seat, or backward (excludes reaching for seatbelt) in the frame
Reading Participant read any printed materials while driving, including receipts, checkbooks, and maps in the frame
Eating Participant performed any behavior related to eating while driving, including holding food items in the frame
Drinking Participant performed any behavior related to drinking while driving, including holding cup/mug/bottle in the frame
Grooming Participant performed any behavior related to grooming while driving including touching their face, looking in mirror to make adjustments, shaving, and putting on make-up in the frame
Smoking Participant was smoking while driving in the frame
Interacting with a cell phone Participant interacted with a cell phone while driving, including holding phone up to their face and holding phone in their hand (does not include hands-free cell phone use) in the frame
Open in a new tab

The current study conceptualized distracted driving behavior at two levels. For descriptive purposes, distracted driving behavior was calculated as the number of trips in which a distracted driving behavior took place. Each participant had one value for this variable, and this variable accounted for whether or not a behavior took place but not duration of the behavior. Next, distracted driving behavior was calculated as the proportion of each driving trip in which the distracted driving behavior took place, measured as the number of frames in each trip in which the behavior occurred divided by the total number of frames for that trip. This across-trip measurement of distracted driving behavior accounted for total duration of a trip so that participants could be compared to each other. Each participant had multiple values for this variable depending on the number of driving trips they took during the study period.

2.3.2. Sleep Duration

Participants completed the seven-component Pittsburgh Sleep Quality Index (PSQI; Buysse, Reynolds III, Monk, Berman, & Kupfer, 1989) once at home during the two-week study period in order to assess sleep over the previous month. Of particular interest in the current study was the sleep duration component given prior literature observed that sleep duration is associated with crashes (Connor et al., 2002; Gottlieb et al., 2018). The sleep duration component takes the score of participants’ responses to the question, “During the past month, how many hours of actual sleep did you get at night? (This may be different than the number of hours you spent in bed)”. Participants could write any value in response to the item.

2.4. Data Simulation

As daily sleep duration was not collected in SANDS, we simulated daily sleep values using the rnorm function in R (R Core Team, 2017). To simulate daily sleep values across the two-week study period, we generated a normal distribution of sleep duration values for each participant using their reported average hours of sleep from the PSQI as the distribution mean and a standard deviation of 1. This is a modified approach to the intended PSQI use which assesses sleep over the past month rather than at the daily level. Simulated sleep data were merged with actual distracted driving data collected in SANDS. For every actual driving trip containing an actual proportion of trip spent engaging in distracted driving behavior, participants also had a value of simulated sleep duration from the previous night. As daily sleep duration data were simulated independently from the distracted driving behavior data, there should not be any significant associations between the two variables which are intended for demonstration purposes.

2.5. Analytic Strategy

All analyses were conducted in SPSS. The first aim to characterize distracted driving behaviors was examined by calculating frequencies of participants who engaged in each distracted driving behavior at all across trips. We also calculated means, standard deviations, and ranges of each distracted driving behavior. We further describe distracted driving behavior by calculating the percent of between- versus within-person variability as an important descriptive statistic. We first ran an intercept-only multilevel model, i.e., unconditional means model, with a random intercept and no predictors to identify the amount of between- and within-person variation in three distracted driving behaviors, quantified by an intraclass correlation coefficient (ICC). The multilevel model for Aim 1 is represented by the following equations:

Level 1:

Yti=β0i+β1i+eti

Level 2:

β0i=γ00+u0iβ1i=γ10

In the above equation, Yti represents the outcome (interacting with cell phone, grooming, or reaching) on trip t for person i. β0i represents the intercept with a random component u0i, and β1i represents the slope which is constant for all participants. Random within-person variability/error is represented by eti. Due to low frequencies of reading, eating, drinking, and smoking, multilevel models could not converge and those specific distracted driving behaviors were removed from subsequent analyses, leaving interacting with a cell phone, grooming, and reaching.

To achieve the second aim, to demonstrate the use of within-person characteristics in predicting driving behavior using a simulated dataset, we added three parameters: time, between-person sleep, and within-person sleep. Sleep duration was centered at the within-person level in the simulated dataset by subtracting each simulated sleep duration from each person’s average sleep duration as reported in the PSQI. The value for daily sleep represented how much each person deviated from their own typical sleep duration on a particular day. Sleep duration was also centered at the between-person level by subtracting each person’s average sleep from the sample average sleep (6.9 hours) to represent how much each person deviated from the sample average sleep duration. The multilevel model for Aim 2 is represented by the following equations:

Level 1:

Yti=β0i+β1i(DAILYSLEEPti)+eti

Level 2:

β0i=γ00+γ01(MEANSLEEPi)+u0iβ1i=γ10

In the above equation, the intercept β0i is determined by a grand mean for distracted driving duration plus between-person differences in sleep duration plus between-person variability. The slope β1i is determined by within-person differences in daily sleep duration, γ10. In total, three separate multilevel models were conducted to model each distracted driving behavior. Multilevel model analyses were conducted using the MIXED function in SPSS 26.

3. Results

3.1. Participants

All participants completed a baseline assessment and a post-test assessment two weeks later and had N-DAD data for at least one trip (n = 68). Participants ranged in age from 65 – 85 years (M = 72.4, SD = 5.4). The majority of participants were highly educated and identified as Caucasian. The participant sample consisted of approximately equal numbers of men and women (see Table 2 for demographics). At baseline, participants self-reported normally driving an average of 6.2 days per week (SD = 1.3).

Table 2.

Demographic characteristics of analytic sample (N=68)

M (SD) N (%) Range
Gender (% women) . 36 (52.9%) .
Race (% White) . 57 (83.8%) .
Race (% Black) .   9 (13.2%) .
Marital status (% married) . 41 (60.3%) .
Years of Education 14.6 (2.5) . 9 – 20
Age in years 72.4 (5.4) . 65 – 85
Reported days/week normally drive at baseline   6.2 (1.3) . 2 – 7
Average hours of sleep/night in past month   6.9 (1.6) . 2.5 – 11
Open in a new tab

3.2. Trips

N-DAD data revealed that the 68 participants took a total of 219 driving trips across two weeks. Available trips per participant ranged from 1 trip to 12 trips, though only a few participants (6%) had more than 8 driving trips available. The duration of each trip was estimated by multiplying the number of frames (taken every 2 seconds) by 2 and then dividing by 60 to obtain an estimate of minutes. Duration was available for 214 trips for a total of 2,429 minutes of driving data available from the sample of 68 drivers with N-DAD data. Trip length was calculated using the GPS data for each trip and was available for 162 driving trips. There was no minimum threshold for trip length, which ranged from 0.04 miles to 26.90 miles. In total, the N-DAD dataset included data for 705.69 miles of driving.

Notably, there was high variability in trip length across participants. Figure 1 displays the variability in trip length for three select participants across 7 trips that had the most variability in trip length based on visual inspection. The solid lines represent the recorded trip length for each driving trip, and the dotted lines represent each participant’s mean trip length across 7 trips. Participant A had a mean trip length of 4.5 miles. Typically, naturalistic driving studies would report the mean trip length or total trip length for each participant, represented by the dotted line. However, Figure 1 shows that Participant A’s trip lengths varied from trip to trip, driving around 11 miles for trips 1 and 3 and then less than 2 miles for trips 4 through 7. Similarly, Participant B drove 10 miles on their first trip but subsequently never drove more than 2 miles until trip 7. If we only looked at Participant B’s mean, we would see that he/she drove an average of 3.21 miles across 7 days and might surmise that he/she may not drive long distances. Finally, the trip length of Participants B and C highlights the importance of examining trip-to-trip variability in driving behavior. The typical approach of testing for mean-level differences in trip length between participants might indicate non-significant differences between Participant B and Participant C. However, the Figure illustrates that there are important trip-to-trip differences in trip length between Participants B and C.

Figure 1.

Figure 1.

Open in a new tab

Trip length in miles for three selected participants and seven trips. Solid lines represent the length for each trip, and dotted lines represent participants’ individual means for trip length.

3.3. Distracted driving behavior

Descriptive statistics of distracted driving behaviors summarized across trips are located in Table 3. In general, distracted driving behaviors were uncommon in this sample. The fourth column on the right reports the percent of participants who engaged in each type of distracted driving behavior at least once over the course of the study. Smoking, reading, eating, and drinking were rare, with less than 20% of the sample ever engaging in these behaviors during the course of the study. A high percentage of participants reached (71%) and/or groomed (43%) while driving at least once during the course of the study; however, the average percent of the driving trip in which these behaviors occurred was small. The second column reports the average percent of a driving trip in which each behavior occurred, and the third column reports the range of percent of a driving trip in which each behavior occurred. The means, standard deviations, and ranges in these columns are calculated from the total number of driving trips for the full sample in which it was possible to determine whether a behavior took place, ranging from 178 trips with reaching data to 182 trips with eating, drinking, and smoking data. All 68 participants contributed to these total trip numbers. For reading, eating, drinking, and smoking, the average percent of a trip in which the behavior occurred was less than 1%, with a small range (from less than 0.5% of a trip to 6.77% of a trip). In general, few participants ever read, ate, drank, or smoked during trips, and participants who did these behaviors did them for a small percentage of driving trip except for a few outliers (such as one participant who groomed for 91.94% of a single driving trip and one participant who smoked for 85.23% of a driving trip).

Table 3.

Descriptive Statistics of Distracted Driving Behaviors in Full Sample (N = 68)

Distracted driving behavior M (SD) % of trip Range % of trip % of participants who ever engaged in behavior
Reaching 1.39 (5.21) 0 – 61.41 70.6%
Reading 0.01 (0.08) 0 – 0.87   8.8%
Eating 0.03 (0.34) 0 – 4.50   8.8%
Drinking 0.15 (0.74) 0 – 6.63 17.6%
Grooming 1.44 (9.54) 0 – 91.94 42.6%
Smoking 0.75 (6.77) 0 – 85.23   5.9%
Interacting with cell phone 1.53 (6.77) 0 – 78.14 26.5%
Open in a new tab

3.4. Across-trip level distracted driving

To further characterize distracted driving behaviors in this sample, intraclass correlation coefficients (ICCs) were calculated for distracted driving behaviors. Model convergence was not achieved for reading, eating, drinking, and smoking due to low frequency of these distracted driving behaviors, so ICCs were only calculated for interacting with a cell phone, reaching, and grooming. Based the ICCs, 93.34% of the variation in reaching and 92.89% of the variation in interacting with a cell phone were due to within-person differences, while only a small percent of reaching (6.66%) and interacting with a cell phone (7.11%) are explained by between-person differences. In other words, participants’ behavior differed more from trip-to-trip than from other participants in reaching and interacting with a cell phone. In contrast, the ICC for grooming was almost 0%, indicating that participants did not differ from one another in their average levels of grooming. This result suggests that almost all of the variability in grooming was due to participants varying from their own usual amount of grooming from trip to trip.

To further illustrate within-person variability in distracted driving behavior in this sample, we selected an additional three participants and three trips that had the most variability based on visual inspection out of 68 participants and 214 trips. Figure 2 displays the proportion of trip spent interacting with a cell phone for a subsample of participants with three consecutive trips with varying proportions of interacting with a cell phone. As can be seen by the solid lines in the figure, proportion of trip spent interacting with a cell phone varied within participants from trip to trip. For example, Participant D did not interact with a cell phone at all during his/her first trip, but spent 4% of trip 2 and 6% of trip 3 interacting with a cell phone. Participant E spent 9% of his/her first trip interacting with a cell phone but then did not interact with a cell phone at all on two subsequent trips. The average time spent interacting on a cell phone for Participant E and F across these three trips (represented by dotted lines) are similar at 3.33% and 3%, but the across-trip assessments of cell phone interaction indicate that these participants’ behavior was quite different across three trips.

Figure 2.

Figure 2.

Open in a new tab

Proportion of trip durations spent interacting with a cell phone for selected three participants and three trips. Solid lines represent the proportion interacting with a cell phone for each trip, and dotted lines represent participants’ individual means for proportion spent interacting with a cell phone.

3.5. Simulation results

Results from the data simulation using mean and daily sleep duration to predict distracted driving behavior are presented in Table 4. As sleep values are randomly simulated, we did not expect a significant association between sleep duration and distracted driving. However, the results demonstrate how components of the multilevel model results can be interpreted to inform driving behavior. In Table 4, the estimate for daily sleep represents the association between daily deviations from average sleep duration and distracted driving behavior. The estimate for mean sleep represents the association between deviation from the sample average hours of sleep reported at baseline and distracted driving behavior. A significant estimate for time would indicate growth across the two-week driving period, though no meaningful trajectories across two weeks were expected. The random effect for intercept indicates whether there is significant variation in intercept, or initial status of distracted driving, between participants.

Table 4.

Multilevel Modeling Results Predicting Distracted Driving from Simulated Daily Sleep Duration (N = 68)

Model Interacting with cell phone Grooming Reaching
Fixed Effects Estimate (SE) Estimate (SE) Estimate (SE)
Intercept −0.01 (0.01) 0.01 (0.02) 0.02 (0.01)
Daily Sleep 0.00 (0.00) 0.00 (0.01) 0.00 (0.00)
Mean Sleep 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
Time 0.01 (0.01) 0.00 (0.00) 0.00 (0.00)
Random Effects
Intercept 0.00 (0.00) 0.01 (0.00) 0.00 (0.00)
Residual 0.00 (0.00) 0.01 (0.00) 0.00 (0.00)
N Observations 181 181 178
N Participants 68 68 68
Open in a new tab

4. Discussion

The present study demonstrated the utility of multilevel modeling in a naturalistic driving dataset, examining distracted driving behaviors in a sample of older adults across two weeks. In this study, a large amount of variation in three distracted driving behaviors (interacting with a cell phone, reaching, and grooming) was due to within-person differences, suggesting that people vary greatly from trip to trip in how they drive. To date, no naturalistic study has reported within-person variability in distracted driving behavior, making this study the first of its kind. By collapsing to number of frames per trip or even a binary variable for whether a behavior occurred during a trip, a large amount of information on within-person differences is lost. Furthermore, aggregating data across trips may inflate the proportion of distracted driving behaviors exhibited while an individual is typically driving. For example, summarizing the number of participants who used a cell phone at least once across the study period may obscure the fact that these participants only did so once over the course of the study period. Roadside observational studies, surveys asking about behaviors on a typical day or averaged across a time frame, and naturalistic observational studies that condense driving behavior data to summary level statistics are not able to capture this variability in distracted driving behavior. An additional advantage to looking at driving behavior across multiple days is that it can provide a more accurate picture of what people’s typical driving behavior looks like. For example, the current study revealed that some participants interacted with a cell phone on one trip but never again in subsequent trips. A roadside observational study like the National Occupant Protection Use Survey would have recorded those participants as having interacted with a cell phone and used this estimate to calculate sample averages of the rate of cell phone use while driving. However, if the survey observed these participants the following day instead, they would be recorded as not using a cell phone while driving, contributing differently to the sample average.

The antecedents of distracted driving behavior likely fluctuate, so it is unsurprising that distracted driving behavior varied so much from trip to trip. Safety-related behaviors should therefore be examined across multiple driving trips to obtain more accurate assessments of their frequency. Some drivers may be driven by motivational factors to use a cell phone during most driving trips, such as using time spent driving to call friends, while other drivers may intentionally not use a cell phone while driving except for rare circumstances. Aside from motivational factors, distracted driving behavior may also be driven by interpersonal factors that a driver may not be aware of, such as the amount of sleep the person got the night before. Insufficient sleep the night before a trip may lead to impaired decision-making that would lead a driver to engage in distractions while driving, though other pathways are possible and can be tested with multilevel modeling approaches. For example, drivers may intentionally refrain from distraction while driving because they are aware of slowed reaction time resulting from getting fewer hours of sleep the previous night. Mediation approaches to structural equation modeling allow for testing of pathways (Preacher, Zhang, & Zyphur, 2011), such as a pathway between sleep, driver intention, and driving behavior. Identification of causal pathways through naturalistic data collection can replace the need for experimentation in understanding these pathways within a person, which can be expensive, time-consuming (as in the case of sleep deprivation studies of driving simulator performance), or even impossible (as in the case of cognition which cannot be experimentally manipulated).

The current study simulated daily sleep data for a sample of older drivers to demonstrate the utility of daily measures in understanding driving behavior. An advantage of simulation is that it enabled us to demonstrate uses of such variability modeling given daily sleep data within a naturalistic driving study were not available. However, the simulated data have limitations as we cannot conclude meaningful associations between sleep and driving. Naturalistic driving studies can collect daily sleep measures by incorporating smartphone or other daily assessments such as wearable technologies into the study design. The current study chose the sleep duration variable of the PSQI as a demonstrator of daily interpersonal characteristics because it has been associated with driving safety (Connor et al., 2002; Gottlieb et al., 2018). Importantly, appropriate assessments of sleep should be used that are intended for daily assessment; the PSQI is intended to assess sleep over a longer period of time (i.e., one month). In designing future naturalistic studies, other sleep parameters may also be included in daily assessments using sleep diaries or actigraphy, such as wake-time after sleep onset and number of nighttime wakenings which have shown to be highly variable across nights (Buysse et al., 2010; Dillon et al., 2015).

Naturalistic studies could also incorporate daily assessments of daily driver characteristics that have been used in psychological research. Driver characteristics related to driving mobility and safety, such as stress (Mather, Gorlick, & Lighthall, 2009; Rowden, Matthews, Watson, & Biggs, 2011), mood (Bernstein, DeVito, & Calamia, 2019; Hu, Xie, & Li, 2013), and cognition (Edwards et al., 2008; Emerson et al., 2012; Wood, Anstey, Kerr, Lacherez, & Lord, 2008), have also been shown to fluctuate within and across days. Most notably, daily within-person relationships have been found among stress and cognition in which reaction time is slower on days when people report more stress (Sliwinski, Smyth, Hofer, & Stawski, 2006). SANDS did not have stress variables assessed at baseline so we could not simulate these data, and the paper-and-pencil cognitive tests in SANDS would not lend themselves to smartphone assessments. Including daily assessments of these and other driver characteristics specifically designed for daily smartphone assessment can provide valuable information on complex associations between driver characteristics and driving safety.

There were several limitations of the current study that warrant discussion and lead to future directions for research. Due to technical difficulties and severe weather, the N-DAD system malfunctioned and some participants’ trips were not recorded or were unable to be coded. In some cases, extreme cold caused the N-DAD system to not turn on when the vehicle started, and these trips were not recorded. This may have biased the distracted driving prevalence estimates, as the weather on these trips was different than the trips in which the N-DAD successfully collected data. In addition to missing data due to technical limitations, distracted driving behaviors were not commonly seen in available video data. As a result, we were unable to compare within- and between-person variability in reading, eating, drinking, and smoking. As participants did not take many trips over the course of the study, it is unclear whether participants were not engaging in distracted driving behaviors as part of their typical behavior or whether these behaviors were simply not captured in the limited number of trips available. Finally, some trips had video data available but were missing GPS data due to a lapse in the GPS signal, so trip length was missing for some trips. Trip length was not added to the multilevel models predicting distracted driving behavior but was presented for descriptive purposes only. Technical limitations are not unique to SANDS, and other naturalistic driving studies have documented issues related to GPS and video capture (Blatt et al., 2015; Klauer et al., 2010). Our missing data are comparable to other naturalistic driving studies, such as the study reported by Molnar and colleagues in which 57 out of 220 participants were removed from analysis because they did not have all or most of their naturalistic driving data available (Molnar et al., 2013).

Another intriguing area of future work looking at variability in driving behavior is examining how within-person variability changes over time. The models in the current study did not include a growth component because we did not expect that driving behavior would change systematically across two weeks. Longer naturalistic studies, such as the 100-Car Study, (twelve months; Klauer et al., 2010), SHRP 2 (two years; Blatt et al., 2015), LongROAD (three years; Li et al., 2017), and the Candrive II/Ozcandrive study (four years; Marshall et al., 2013), will allow for examination of longitudinal driving changes and how trip-level variability is associated with longitudinal change in driving behavior. For example, as people become older, they may decrease in their variability of trip length, traveling to fewer places.

Finally, naturalistic data collection methods focused on variability can inform driver safety prevention. Ecological momentary interventions (EMIs) deliver interventions to people on a more frequent basis than traditional interventions and in an everyday context (Heron & Smyth, 2010). A main feature of EMIs is that they can provide real-time feedback to people at the most appropriate times in order to maximize intervention benefit. EMIs for health behavior and symptoms are acceptable to participants and have been shown to be beneficial for treatment of behavior and symptoms (Heron & Smyth, 2010). An EMI approach can be extended from health behavior and psychology into the world of driving safety. A smartphone-based EMI can use daily sleep, stress, and/or cognition to provide feedback to a driver before they take a driving trip, reminding them about the importance of driving without distractions, especially under varying conditions. For example, if a driver reports high stress or poor sleep in the morning, the smartphone can deliver a message before a driving trip later that day reminding them to stay focused behind the wheel. Tailored interventions delivered on a more regular basis may be more effective than traditional public health campaigns aimed at reducing distracted driving.

5. Conclusions

This study is the first to quantify trip-to-trip variability in driving behavior utilizing naturalistic driving data in a sample of older adults and demonstrate the use of a daily driver characteristic as a predictor of this variability. Importantly, these preliminary analyses demonstrate that between 92.89% and 99.99% of variability in distracted driving behaviors are due to within-person differences. That is, participants’ driving differed more from trip to trip than between drivers. This is a novel contribution to the literature and highlights an area in need of further exploration in larger naturalistic datasets. This work can inform analysis of already collected data and planning for future studies. Future naturalistic driving studies should incorporate daily assessments of interpersonal characteristics which can provide important context to driving data and can supplement the need for experimental design in understanding causal pathways. Approaches prioritizing variability in driving behavior can also benefit prevention efforts aimed at improving driving safety. Distracted driving is one example of a modifiable driving behavior strongly related to safety that can be better understood by uncovering not only which drivers are more likely to engage in these behaviors but when a driver is more likely to engage in these behaviors. Within-person perspectives on driving can even be extended to the use of ecological momentary interventions in providing more precise prevention of distracted driving. As demonstrated by this study, traditional naturalistic driving assessments coupled with daily assessments of driver characteristics, analyzed with minimal data reduction to preserve variability and allow for multilevel modeling techniques, provide rich data on driving behavior important for safety.

Highlights.

  • We demonstrate the use of multilevel modeling to examine variability in driving behaviors.

  • Distracted driving varied within and between drivers in a naturalistic driving study.

  • Cell phone use and reaching behaviors varied more across trips than across people.

  • Simulated daily sleep demonstrates how predictors of driving can vary within a person.

Acknowledgments

Funding:

This work was supported by the Southeastern Transportation Research, Innovation, Development and Education (STRIDE) Center; Alabama Department of Transportation (ALDOT); Florida Department of Transportation (FDOT), University of Alabama at Birmingham Faculty Development Grant Program, and the Edward R. Roybal Center for Translational Research in Aging and Mobility, National Institute on Aging [2 P30 AG022838]. The funders were not involved in any aspect of conducting of the research or preparation of the article.

Footnotes

Declarations of interest: None

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Sara A. Freed, The Pennsylvania State University, 119 Health and Human Development Building, University Park, PA 16802

Lesley A. Ross, Clemson University, 418 Brackett Hall, Clemson, SC 29634

Alyssa A. Gamaldo, The Pennsylvania State University, 119 Health and Human Development Building, University Park, PA 16802

Despina Stavrinos, The University of Alabama at Birmingham, 916 19th Street South, Birmingham, AL 35294

References

  1. AAA Foundation for Traffic Safety. (2019). 2018 Traffic Safety Culture Index. Washington, DC: Retrieved from https://aaafoundation.org/2018-traffic-safety-culture-index/ [Google Scholar]
  2. Aksan N, Anderson SW, Dawson J, Uc E, & Rizzo M (2015). Cognitive functioning differentially predicts different dimensions of older drivers’ on-road safety. Accident Analysis and Prevention, 75, 236–244. doi: 10.1016/j.aap.2014.12.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aksan N, Dawson JD, Emerson JL, Yu L, Uc EY, Anderson SW, & Rizzo M (2013). Naturalistic distraction and driving safety in older drivers. Human Factors: The Journal of the Human Factors and Ergomanics Society, 55(4), 841–853. doi: 10.1177/0018720812465769 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bernstein JPK, DeVito A, & Calamia M (2019). Associations between emotional symptoms and self-reported aberrant driving behaviors in older adults. Accident Analysis and Prevention, 127, 28–34. doi: 10.1016/j.aap.2019.02.024 [DOI] [PubMed] [Google Scholar]
  5. Blatt A, Pierowicz J, Flanigan M, Lin P-S, Kourtellis A, Jovanis P, … Hoover M (2015). Naturalistic driving study: Field data collection. Retrieved from Washington, DC. [Google Scholar]
  6. Buysse DJ, Reynolds CF III, Monk TH, Berman SR, & Kupfer DJ (1989). The Pittsburgh Sleep Quality Index: A new instrument for psychiatric practice and research. Psychiatric Research, 28(2), 193–213. doi: 10.1016/0165-1781(89)90047-4 [DOI] [PubMed] [Google Scholar]
  7. Buysse DJ, Cheng Y, Germain A, Moul DE, Franzen PL, Fletcher M, & Monk TH (2010). Night-to-night sleep variability in older adults with and without chronic insomnia. Sleep medicine, 11(1), 56–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Choudhary P, & Velaga NR (2017). Analysis of vehicle-based lateral performance measures during distracted driving due to phone use. Transportation Research Part F, 44, 120–133. doi: 10.1016/j.trf.2016.11.002 [DOI] [Google Scholar]
  9. Connor J, Norton R, Ameratunga S, Robinson E, Civil I, Dunn R, … Jackson R (2002). Driver sleepiness and risk of serious injury to car occupants: Population based case control study. BMJ, 324(7346), 1–5. doi: 10.1136/bmj.324.7346.1125 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cross JM, McGwin G Jr, Rubin GS, Ball KK, West SK, Roenker DL, & Owsley C (2009). Visual and medical risk factors for motor vehicle collision involvement among older drivers. The British Journal of Ophthalmology, 93(3), 400–404. doi: 10.1136/bjo.2008.144584 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dillon HR, Lichstein KL, Dautovich ND, Taylor DJ, Riedel BW, Variability in self-reported normal sleep across the adult age span. The Journals of Gerontology: Series B, 70(1), 46–56. 10.1093/geronb/gbu035 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Edwards JD, Ross LA, Ackerman ML, Small BJ, Ball KK, Bradley S, & Dodson JE (2008). Longitudinal predictors of driving cessation among older adults from the ACTIVE clinical trial. Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 63(1), P6–P12. doi: 10.1093/geronb/63.1.P6 [DOI] [PubMed] [Google Scholar]
  13. Ellison AB, Greaves SP, & Bliemer MCJ (2015). Driver behaviour profiles for road safety analysis. Accident Analysis and Prevention, 76, 118–132. doi: 10.1016/j.aap.2015.01.009 [DOI] [PubMed] [Google Scholar]
  14. Emerson JL, Johnson AM, Dawson JD, Uc EY, Anderson SW, & Rizzo M (2012). Predictors of driving outcomes in advancing age. Psychology and Aging, 27(3), 550–559. doi: 10.1037/a0026359 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Ferdinand AO, & Menachemi N (2014). Associations between driving performance and engaging in secondary tasks: A systematic review. American Journal of Public Health, 104(3), 39–48. doi: 10.2105/AJPH.2013.301750 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Fofanova J, & Vollrath M (2011). Distraction while driving: The case of older drivers. Transportation Research Part F: Traffic Psychology and Behavior, 14(6), 638–648. doi: 10.1016/j.trf.2011.08.005 [DOI] [Google Scholar]
  17. Fragoso CAV, Araujo KLB, Van Ness PH, & Marottoli RA (2008). Prevalence of sleep disturbances in a cohort of older drivers. Journals of Gerontology Series A: Biological Science and Medical Science, 63A(7), 715–723. doi: 10.1111/jgs.12454 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Friedman C, McGwin G Jr, Ball KK, & Owsley C (2013). Association between higher order visual processing abilities and a history of motor vehicle collision involvement by drivers ages 70 and over. 54(1), 778–782. doi: 10.1167/iovs.12-11249 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gamaldo AA, Allaire JC, & Whitfield KE (2010). Exploring the within-person coupling of sleep and cognition in older African Americans. Psychology and Aging, 25(4), 851–857. doi: 10.1037/a0021378. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gottlieb DJ, Ellenbogen JM, Bianchi MT, & Czeisler CA (2018). Sleep deficiency and motor vehicle crash risk in the general population: A prospective cohort study. BMC Medicine, 16(44), 1–10. doi: 10.1186/s12916-018-1025-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Heron KE, & Smyth JM (2010). Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. British journal of health psychology, 15(1), 1–39 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hu T-Y, Xie X, & Li J (2013). Negative or positive? The effect of emotion and mood on risky driving. Transportation Research Part F, 16, 29–40. doi: 10.1016/j.trf.2012.08.009 [DOI] [Google Scholar]
  23. Huisingh C, Levitan EB, Irvin MR, MacLennan P, Wadley V, & Owsley C (2017). Visual sensory and visual-cognitive function and rate of crash and near-crash involvement among older drivers using naturalistic driving data. Investigative Opthalmology and Visual Science, 58(7), 2959–2967. doi: 10.1167/iovs.17-21482 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Klauer SG, Guo F, Simons-Morton BG, Ouimet MC, Lee SE, & Dingus TA (2013). Distracted driving and risk of road crashes among novice and experienced drivers. The New England Journal of Medicine, 370(1), 54–59. doi: 10.1056/NEJMsa1204142 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Klauer SG, Guo F, Sudweeks J, & Dingus TA (2010). An analysis of driver inattention using a case-crossover approach on 100-car data: Final report. (DOT HS 811 334). Washington, DC: National Highway Traffic Safety Administration; Retrieved from http://www.nhtsa.gov.edgesuite-staging.net/Research/Crash+Avoidance/ci.Distraction.print [Google Scholar]
  26. Li G, Eby DW, Santos R, Mielenz TJ, Molnar LJ, Strogatz D, … LongROAD Research Team. (2017). Longitudinal research on aging drivers (LongROAD): Study design and methods. Injury Epidemiology, 4(22), 1–16. doi: 10.1186/s40621-017-0121-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Mather M, Gorlick MA, & Lighthall NR (2009). To brake or accelerate when the light turns yellow? Stress reduces older adults’ risk taking in a driving game. Psychological Science, 20(2), 174–176. doi: 10.1111/j.1467-9280.2009.02275.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Molnar LJ, Charlton JL, Eby DW, Bogard SE, Langford J, Koppel S, … Man-SonHing M (2013). Self-regulation of driving by older adults: Comparison of self-report and objective driving data. Transportation Research Part F: Traffic Psychology and Behavior, 20, 29–38. doi: 10.1016/j.trf.2013.05.001 [DOI] [Google Scholar]
  29. Molnar LJ, Eby DW, Bogard SE, LeBlanc DJ, & Zakrajsek JS (2018). Using naturalistic driving data to better understand the driving exposure and patterns of older drivers. Traffic Injury Prevention, 19, S83–S88. doi: 10.1080/15389588.2017.1379601 [DOI] [PubMed] [Google Scholar]
  30. National Highway Traffic Safety Administration. (2018). Distracted Driving 2016. (DOT HS 812 517). Washington, DC: National Center for Statistics and Analysis [Google Scholar]
  31. Preacher KJ, Zhang Z, & Zyphur MJ (2011). Alternative methods for assessing mediation in multilevel data: The advantages of multilevel SEM. Structural Equation Modeling, 18(2), 161–182. doi: 10.1080/10705511.2011.557329 [DOI] [Google Scholar]
  32. R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/ [Google Scholar]
  33. Rowden P, Matthews G, Watson B, & Biggs H (2011). The relative impact of work-related stress, life stress and driving environment stress on driving outcomes. Accident Analysis and Prevention, 43(4), 1332–1340. doi: 10.1016/j.aap.2011.02.004 [DOI] [PubMed] [Google Scholar]
  34. Schneidereit T, Petzoldt T, Keinath A, & Krems JF (2017). Using SHRP 2 naturalistic driving data to assess drivers’ speed choice while being engaged in different secondary tasks. Journal of Safety Research, 62, 33–42. doi: 10.1016/j.jsr.2017.04.004 [DOI] [PubMed] [Google Scholar]
  35. Singer JD, & Willett JB (2003). Applied longitudinal data analysis: Modeling change and event occurrence. NY: Oxford University Press. [Google Scholar]
  36. Sliwinski MJ, Smyth JM, Hofer SM, & Stawski RS (2006). Intraindividual coupling of daily stress and cognition. Psychology and Aging, 21(3), 545–557. doi: 10.1037/0882-7974.21.3.545 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Stavrinos D, Ross L, & Sisiopiku V (2014). Final Report: A Naturalistic Driving Study Across the Lifespan (2012–095S). Retrieved from Gainesville, FL: https://stride.ce.ufl.edu/wp-content/uploads/2017/03/Stavrinos_Ross_STRIDE_Final-Report.pdf [Google Scholar]
  38. Thompson KR, Johnson AM, Emerson JL, Dawson JD, Boer ER, & Rizzo M (2012). Distracted driving in elderly and middle-aged drivers. Accident Analysis and Prevention, 45(1), 711–717. doi: 10.1016/j.aap.2011.09.040 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Wood JM, Anstey KJ, Kerr GK, Lacherez PF, & Lord S (2008). A multidomain approach for predicting older driver safety under in-traffic road conditions. Journal of the American Geriatrics Society, 56(6), 986–993. doi: 10.1111/j.1532-5415.2008.01709 [DOI] [PubMed] [Google Scholar]

RESOURCES