A Methodological Approach to Retaining Accelerometers and Associated Data in Community-Based Studies

Abstract

Recent technological advances allow for field-based data collection of accelerometers in community-based studies. Mail-based administration can markedly reduce the cost and logistic challenges and burden associated with in-person data collection. It necessitates, however, other resources, such as phone calls and mailed reminder prompts, to increase protocol compliance and data recovery. Additionally, lost accelerometers can impact the study’s budget and its internal validity due to missing data. In this article, we present an applied methodological approach used to define thresholds (or cutoff points) at which pursuing unreturned accelerometers is a worthwhile versus futile pursuit. This methodological approach was designed, specifically, to maximize scalability across multiple sectors. We used data from an on-going study that administered accelerometers through the mail to illustrate and encourage investigators to replicate the approach for use in their own studies. In heterogeneous study samples, investigators might consider repeating this approach by study-relevant strata to refine thresholds and improve the return percentages of data collection instruments, minimize the potential missing data, and optimize study staff time and resources.

The U.S. Postal Service has been used extensively to deliver and receive health-related questionnaires for public health surveillance and other research purposes (Edwards et al., 2002; Griffis, Goldsby, & Cooper, 2003; Holen et al., 2016; Mavis & Brocato, 1998; Sakhi et al., 2015). However, recent technological advances now allow for mail-based delivery and collection of behavioral measures (e.g., accelerometry for physical activity assessment) in large epidemiological studies (Lee & Shiroma, 2014). For example, beginning in the early 2000s, the Reasons for Geographic and Racial Differences in Stroke (REGARDS) Study (Howard et al., 2015), British Regional Heart Study (Lee & Shiroma, 2014), Woman’s Health Study (Shiroma, Freedson, Trost, & Lee, 2013), and Woman’s Health Initiative Study (WHI) (UNC Center for Health Promotion and Disease Prevention, 2017), among others, have provided accelerometers to participants by mail. Postal service administration of data collection materials can reduce costs, participant burden, and logistic challenges associated with in-person data collection, yet it necessitates allocation of other resources to optimize protocol compliance and data recovery (Edwards et al., 2002; Kelley, Clark, Brown, & Sitzia, 2003). Generally, this has consisted of monetary incentives, reminder phone calls and postcards, or pick-up appointments at the participants home or community center. Providing pre-stamped return materials and using recorded delivery services also improves return rates (Edwards et al., 2002). Additionally, unreturned or lost materials can have a significant budgetary impact on research studies (Griffis et al., 2003; Mavis & Brocato, 1998). Although accelerometers are now relatively inexpensive (costing $250 to $350 each), when accounting for the number of devices needed to make up for attrition, this can result in a very large proportion of a study’s budget. For example, in the REGARDS Study, a relatively low proportion (8.0%) of accelerometers were not returned, yet due to the large study sample (n = 12,146), this resulted in a budgetary impact of approximately $265,000 (Howard et al., 2015). In other studies using the mail to deliver and receive accelerometers, the loss rates ranged from 2.1% to 10% (Hansen, Kolle, Dyrstad, Holme, & Anderssen, 2012; Lee & Shiroma, 2014). In addition to budgetary concerns, timely return of accelerometers is now of even greater concern when using the postal service to deliver and retrieve the devices considering the threat of missing data from limited battery life for data acquisition and storage. Hence, optimizing return percentages is critical to appropriately allocate study staff and resources to optimize protocol compliance, as well as for retrieving the maximum amount of valid, high-quality data collected.

In this study, we present an easy to implement approach to establish thresholds of time, at which point pursuing unreturned postal service administered accelerometers can be considered ineffectual. A supplemental file is provided that allows investigators to input data from their own study to create study specific thresholds of time which can then be used to implement more aggressive follow-up techniques (examples provided herein), or to cease pursuit of data collection materials. To illustrate this approach and to demonstrate its utility, we used data from an ongoing community-based study to identify participant profiles with varying protocol compliance and thresholds for outstanding accelerometer pursuit.

Methodology

For the purposes of this study, an outstanding accelerometer is defined as one that has been sent to a participant and has yet to be returned. This approach aims to accomplish two overall goals. The first is to determine how long it takes for an accelerometer to be returned. The second is to determine thresholds of days after which the pursuit of an outstanding accelerometer is futile due to a very low probability of return. It is important to note that these two results are study specific and will differ based on a number of factors (e.g., participant characteristics, data collection protocol, participant compensation, etc.). Table 1 outlines three basic steps to accomplish the overall goals, including the procedures for each step.

Table 1.

Steps and Procedures to Estimate How Many Accelerometers Will be Returned in Mail-based Studies

Step	Procedure
1. Identify compliant participants	a. Identify participants who returned an accelerometer. b. Of participants who returned an accelerometer, identify those who provided valid and invalid data. c. Define compliant participants based on the research question.*
2. Plot the days to return an accelerometer against quantiles of uniform distribution.	a. Calculate the time in the chosen metric (hours, days, weeks, months) between the accelerometer sent and received (time outstanding) for each participant. b. Rank the time outstanding, starting with 1 for the least amount of time, to kth for the greatest amount of time for each participant. c. Calculate the percentiles of values in time outstanding for each participant.
3. Estimate the thresholds of time to return an outstanding accelerometer.	a. Based on the time outstanding, calculate the kth percentile of values. b. Select the percentiles of interest based on the research question at hand. c. Apply the return percentile.exc function to the range of time outstanding for each percentile of interest from 3.**

^*If only interested in retrieving valid data, compliant participants would be defined as any participant returning an accelerometer with valid wear-time. Alternatively, if interested in retrieving all accelerometers (with and without valid data), compliant participants would be defined as any participant returning an accelerometer with or without valid wear-time.

^**The return probabilities are calculated by taking the inverse of the percentile rank. For example, for the 95th percentile, the return probability is 1 – 0.95 = 0.05.

Step 1 is to identify compliant participants, or those participants that have returned an accelerometer to the study site. Depending on the research question, a compliant participant could be defined as a participant that returns an accelerometer with valid data, or a participant that returns an accelerometer with or without valid data. After defining a compliant participant, Step 2 is to determine the number of days an accelerometer was outstanding for each compliant participant. The number of days outstanding for complaint participants can then be plotted and placed into percentiles. Step 3 is to plot the ordered values of the days to return an accelerometer against quantiles of uniform distribution. This is done to establish thresholds or cutoffs in days after an accelerometer has been sent that indicate critical points in time when the probability of accelerometer return is diminished. These cutoffs can be used as a prompt to initiate more intensive retrieval protocols to improve overall return percentage, or to cease retrieval protocols, to more efficiently direct research personnel time and efforts.

Online Supplement

The programming and application of this approach, with imbedded code, is provided in the Online Supplementary Material (S1 Tables [available online]). Instructions are provided within the spreadsheet to help the reader apply this methodology to their data.

Application of Approach

Data from the Houston Travel Related Activity in Neighborhoods (TRAIN) Study were used to demonstrate the procedures outlined above, and explain the utility of this approach for optimizing data collection protocols. Briefly, the TRAIN Study is a prospective, longitudinal evaluation of the effects of a public mass transit expansion on physical activity. For a complete description of the TRAIN Study methodology, see Durand et al. (2016). TRAIN Study participant characteristics are explained in detail elsewhere (Knell et al., 2017). Briefly, participants were primarily non-Hispanic black (37.8%) and Hispanic or Latino (23.6%), female (58.1%), aged 45–64 years (47.3%) or 65 years or older (21.4%), classified as overweight or obese (66%), and had at least a high school education (76.3%). For the purposes of this study (i.e., illustrate this approach), only data from the first 15-months of the baseline data collection period (21-months) were included in the analysis. The Houston TRAIN Study protocol and this secondary analysis were reviewed and approved by The University of Texas Health Science Center (UTHealth) at Houston Committee for the Protection of Human Subjects (UTHealth IRB). Informed consent was obtained from all participants.

In the TRAIN Study, physical activity was measured via questionnaire and accelerometers. Participants were sent the accelerometer in the mail, and asked to return it in the mail after wearing the device over a 7-day period. A prepaid, preaddressed envelope for sending the accelerometer back to the study site was provided to the participant. The mailing transit time between the study site and study coordinating center was 2–3 business days. Therefore, the total transit time (time between sending and receiving materials from the study coordinating center) for a fully compliant participant (i.e., immediately began wearing the accelerometer upon receipt, and immediately returned the accelerometer upon completion) is approximately 14 days. However, after 12 months of data collection, we estimated the total transit time to be a median of 31 days. This discrepancy between the estimated total transit time for compliant participants and what was observed necessitated an approach that: 1) Estimates how many accelerometers we expect to receive back after any number of days outstanding; and 2) Determines critical time points at which the probability of return is diminished greatly so that we could implement more direct efforts at accelerometer retrieval.

Step 1: Identify Compliant Participants.

Accelerometers were sent continuously (typically as weekly batches) to newly enrolled participants. In the first 15-months of data collection, 381 accelerometers were sent to participants. The accelerometers used in the Houston TRAIN Study cost approximately $275 each, and therefore a relatively small proportion (e.g., 10%) of lost materials could have a significant budgetary impact ($10,477). As such, in this case, ‘compliant participants’ were defined as participants who returned an accelerometer, with or without valid data. However, this same methodology can be used in other studies restricting the analysis of ‘compliant participants’ to those returning the data collection instrument with valid data. For example, saliva sampling costs approximately $20 each for the data collection instrument (not including postage). The lower cost, and the fact that the data collection instrument cannot be reused for other participants, would suggest that this analysis should only include those data collection instruments returned with valid data. In the TRAIN Study, of the 381 accelerometers sent to participants, 302 (79%) had been returned within the first 15-months of data collection, and 247 (64.8%) had valid data.

Step 2: Determine the Number of Days to Return an Accelerometer.

In the TRAIN study, we observed that the mean number of days to return an accelerometer (n = 302) was 30.1 days, with a standard deviation of 26.2, and a range of 9–273 days. Figure 1 is a plot of the ordered values of days to return an accelerometer against quantiles of a uniform distribution among those returned in the first 15-months of data collection.

Figure 1 —

Distribution of days to return an accelerometer among compliant participants, TRAIN Study, 2013–2015. Note: Compliant participants are those that returned an accelerometer regardless of validity of data.

Step 3: Estimate the Thresholds of Time to Return an Outstanding-Accelerometer.

This plot, and the thresholds of time to return an accelerometer, can be useful for establishing alerts based on the time elapsed since an accelerometer was sent. The proposed thresholds were set at 5th, 10th, 25th, 50th, 75th, 90th, 95th, and 99th percentiles. In the case of the TRAIN study, the days to return an accelerometer were 14, 15, 19, 24, 34, 48, 62, and 184, respectively. This plot and the respective thresholds indicate that few accelerometers were returned after ∼60 days, and even fewer after ∼120 days (95th percentile). Hence, in this case, at the 60-day mark a more persistent approach should be taken to retrieve the accelerometers (e.g., telephone calls, email, letters) until the 120-day mark is reached, at which time the more persistent approach is likely futile.

Establishing Profiles of Compliant Participants.

As previously mentioned, the thresholds found in the TRAIN Study are specific for the study population, and likely dependent upon a number of factors that may influence accelerometer return percentages and rates. Therefore, for demonstration purposes, we stratified the data by two factors, the sex of participants and participant recruitment mode; that is, participants recruited by a third-party market research firm versus participants recruited by the internal research staff. This subgroup analysis was done to further understand accelerometer return distributions and establish specific cutoffs points for accelerometer pursuit (i.e., probability of return <5%) is no longer worthwhile (see Figure 2). Findings suggest differences in the distribution and cutoffs for accelerometer pursuit varied by recruitment strategy. The 95th percentile among those recruited by the third-party was 68 days, whereas those recruited by internal staff was 49 days. Similarly, the 95th percentile for men was 71 days in comparison to 59 days for women. Thus, for participants who were women or were recruited by internal staff, we should be concerned about outstanding accelerometers at a 50-day time point. Alternatively, for those recruited by the third party or for men, there would be a ∼20% probability to retrieve an outstanding accelerometer after 50 days. Hence, the initiation of follow-up protocols could be based on these specific cutoff points and specific recruitment strategies could be designed based on participants’ characteristics.

Figure 2 —

Distribution of days to accelerometer return by recruitment mode and sex, TRAIN Study, 2013–2015. ^a Recruitment mode was 3rd party firm. ^b Recruitment mode was internal staff. ^c Male participants. ^d Female participants.

Follow-up Actions to Improve Accelerometer Return.

As others have pointed out, best-practices for any study utilizing self-administered data collection measures are systematic, regular, and personalized follow-ups with participants (Edwards et al., 2002). In the TRAIN Study, the follow-up protocol included a follow-up telephone phone call during the data collection period and reminder telephone calls and postcards prompting the return of accelerometers and data collection materials. The follow-up phone call was made to participants 1-week following the accelerometer being sent, approximately 3–5 days into the 7-day data collection period. The purpose of the phone call was to ensure receipt of the accelerometer, confirm the participant was wearing the accelerometer, answer any questions the participant may have regarding accelerometer protocol, trouble-shoot any problems, and to establish a personal connection with the participant. If, during this phone call, the participant reported that s/he hadn’t started to wear the accelerometer, but there was sufficient battery life to collect seven days of accelerometer data, s/he was instructed to begin wearing the accelerometer immediately. Concurrently, a postcard was sent to participants that reminded them to follow the accelerometer protocols and to contact study staff with any questions. At 2-weeks following the accelerometer being sent, a second phone call was made to participants if the first phone call was unsuccessful (did not speak to the participant). Additionally, a second postcard was sent reminding the participant to return the accelerometer to the study site.

Based on the threshold established as described herein, if the accelerometer was not retuned within 60 days, more aggressive follow-up measures were initiated. More regular phone calls were made to those participants with an outstanding accelerometer. Study staff 1) reminded participants that compensation for participation was contingent upon receipt of the accelerometer, and 2) offered additional materials to mail back the instrument. Additionally, if efforts to reach the participant were unsuccessful, we contacted the person listed by the participant to contact should we have trouble reaching them. Finally, after a period of 120 days (∼95th percentile) post the initial mailing of the accelerometer, any directed efforts to retrieve the accelerometer from participants ceased. These elements of the protocol were only put into practice after 15-months of data collection, when the 60- and 120-day thresholds were determined. As this was a longitudinal study, subsequent years of follow-up included contacting the participants regardless of the status of their accelerometer return at baseline. At times, these contacts would result in dialogue concerning the status of the outstanding accelerometer wherein the study staff would encourage the participant to return the accelerometer. Unfortunately, the TRAIN Study was not designed a priori to evaluate the effectiveness of various methods to retrieve accelerometers from the field. Therefore, we are unable to determine which method, if any, performed the best or if all methods performed better than none. Yet, these are interesting and important questions for future research.

Conclusion

The current study presents an approach to establish cut-off points at which pursuing unreturned data collection accelerometers is a worthwhile or futile pursuit in field studies, for the purposes of reducing time and monetary costs associated with outstanding accelerometer pursuit, and for minimizing missing data. We utilized data from the TRAIN Study to demonstrate the applicability of this approach to community-based studies. Based on the results of the analysis presented herein from the TRAIN Study, the 60-day mark initiated more intensive follow-up measures to retrieve outstanding data collection accelerometers from the field. At the 120-day mark, these efforts ceased based on the low likelihood of receiving back any accelerometers. These highly specific and empirically based thresholds allowed the TRAIN Study resources to be used more efficiently and effectively. As such, the TRAIN Study example and results are not intended to present results for comparability or applicability to any future studies. As demonstrated in Figure 2, return thresholds are likely to differ, perhaps drastically, by participant characteristics and other factors that may influence willingness to comply with study protocols. The results from the TRAIN Study should only be used as a demonstration of the methods described in Table 1. While this approach warrants further investigation, current findings show promise and potential to be integrated into future community-based studies to optimize data collection and the appropriate allocation of resources. This method can be appropriately scaled to accommodate small, cross-sectional studies to large, prospective cohort studies that utilize mail-based data collection.

To account for the variability in return time by study and participant characteristics, we demonstrated how the approach to determine the thresholds for outstanding data collection material pursuit can be used to establish profiles of compliant participants. Future studies should consider replicating these methods, using their own data, and should consider factors that contribute to variability in participant compliance. Previous work in mail-based survey research has found that incentives, education level, sex, age, survey length, personalized surveys, question type, anonymity, and survey and font color can have an effect on response rates (Edwards et al., 2002; Griffis et al., 2003; Kelley et al., 2003; Mavis & Brocato, 1998). These factors, and any elements of the protocol that changed at any point during the course of the study, should be considered. For instance, as described in the TRAIN Study, there was a change in recruitment method from a 3rd party firm to internal, and often times, face-to-face recruitment strategies. We hypothesized that the more personalized mode during the second phase of recruitment would produce greater protocol compliance than the strictly telephone-based recruitment employed by a 3rd party firm. This hypotheses was confirmed by the findings shown in Figure 2. Findings suggest differences in the distribution and cutoffs for accelerometer pursuit varied by recruitment strategy and could therefore provide personalized points in time for to initiate more aggressive follow-up protocols. Although this strategy was not undertaken in the TRAIN Study, it may have been more effective, in terms of retrieving outstanding accelerometers, to initiate more aggressive follow-up (more frequent phone calls to participants) to female participants and those recruited by internal staff at 50-days, where there was a 5% probability to retrieve an outstanding accelerometer. Perhaps more importantly, would be the personalized points in time to cease follow-up and save study resources. In the case of the TRAIN Study, one could make the determination that continued follow-up after the 99th percentile threshold is reached is futile, and thereby institute a stoppage of pursuit at 263-days for those recruited by a 3rd party firm, versus 71-days for those recruited by internal staff.

It should be noted that the methodological approach described in the current study utilizes baseline (cross-sectional) data from a prospective study. Thus, future studies using a prospective design should consider repeating these procedures at regular intervals to account for attrition rates. As a prospective study continues data collection with previously compliant participants, the probability of returning an accelerometer among the study sample might improve and therefore the percentiles of days to return and accelerometer may change. Furthermore, estimation of future accelerometer return requires a sufficient sample of returned accelerometers to obtain a reasonable accelerometer return prediction. However, it could be argued that the described methodology could be implemented at any point in time when there are available data from compliant participants, and subsequently repeated and refined until there is no observed improvement upon the accuracy of the prediction. Additionally, while the presented methodological approach to optimize data collection warrants testing in additional study designs and samples, it is likely applicable to any setting and can be replicated within one’s own source population.

In summary, studies using mail-based data collection protocols would greatly benefit from being able to forecast the return percentages of data collection instruments. More importantly is the ability to determine critical points in time when more persistent participant follow-up protocols for instrument retrieval are warranted, and when these measures are futile. This information is critical for community-based studies since they are “resource intensive” in terms of required human and monetary resources needed to conduct studies in the field. Thus, improving the return percentages of instruments, will yield the best “bang for the buck” in study resources, thereby leading to a more cost-effective approach. As research funds are limited, the need to demonstrate cost-effective research operations to grant officers becomes paramount. To this end, this study presents an applied approach to reduce missing data, optimize study resources, and minimize lost data collection instruments.