Minimizing Attrition for Multi-Site Emergency Care Research

Abstract

Loss to follow-up of enrolled patients (aka attrition) is a major threat to study validity and power. Minimizing attrition can be challenging even under ideal research conditions, including the presence of adequate funding, experienced study personnel, and a refined research infrastructure. Emergency care research is shifting towards enrollment through multi-site networks, but there have been limited descriptions of approaches to minimize attrition for these multi-center emergency care studies. This concept paper describes a stepwise approach to minimize attrition, using a case example of a multi-site emergency department prospective cohort of over 3,000 patients that has achieved a 30-day direct phone follow-up attrition rate of < 3%. The seven areas of approach to minimize attrition in this study focused on patient selection, baseline contact data collection, patient incentives, patient tracking, central phone banks, local enrollment site assistance and continuous performance monitoring. Appropriate study design, including consideration of these methods to reduce attrition, will be time well spent and may improve study validity.

Introduction

The Institute of Medicine and the National Institutes of Health (NIH) have highlighted the importance of emergency care research to improve the medical care delivered to acutely ill patients.^1–4 Emergency care clinical research should ideally be conducted at multiple sites to improve the sample size, the speed at which the sample size can be attained, and generalizability of the findings. Several emergency care research networks exist to facilitate such work, and investigators often create ad hoc networks specifically for certain studies.⁵

Minimizing loss to follow-up of enrolled patients, or attrition, may be particularly challenging for multi-site emergency care research. Variations in patient populations, research staff, and protocol implementation among enrolling sites may complicate attempts to achieve high follow-up rates. Prior review articles have summarized predictors of attrition in emergency care research or described procedures to reduce attrition for single site studies.^5,6 To our knowledge, no prior report has described an approach to minimize attrition for multi-site emergency care research.

In this paper we describe the methods of our approach to reducing attrition for multi-site emergency care research. We illustrate these features with an ongoing, 11-site prospective cohort study that has achieved a 2.6% attrition rate at 30-day direct phone follow-up. Emergency care researchers who wish to design and implement investigator-initiated, multi-site studies should consider this approach to minimize subject attrition.

Why Does Attrition Matter?

Attrition threatens study validity and effective statistical power.⁷ Differential attrition by intervention assignment, illness severity, or other patient characteristics may introduce bias into the study results. For example, if side effects associated with an intervention made it more difficult to comply with follow-up procedures, the resulting increase in attrition would make the intervention appear less dangerous than it actually is. The potential for bias increases with the magnitude of attrition, and the direction of bias is often unknown.

Consider a hypothetical randomized trial of 200 patients allocated equally to treatments “A” and “B.” Assume that there is a 30% attrition rate in both arms, and that no primary outcomes are detected in patients who completed follow-up. All of the following scenarios are consistent with the data: “A” has a 30% higher primary outcome rate than “B”; “B” has a 30% higher primary outcome rate than “A”; and “A” and “B” have equivalent primary outcome rates. This example demonstrates the difficulty of drawing meaningful conclusions when the attrition rate is high.

Even in select cases where it is reasonable to assume minimal bias, attrition will reduce effective sample size and statistical power to detect a meaningful outcome difference. Consider the prior example of the randomized trial of “A” vs “B.” Assuming that loss-to-follow-up occurs completely at random, attrition in this scenario will diminish statistical power by 30%. Additionally, should a rare but important outcome (i.e. death of a patient with low-risk chest pain) occur in a patient lost to follow-up, this could potentially change the overall analysis or conclusions.

Several factors may mitigate the impact of attrition on study validity. The potential for bias may be reduced if measurable factors are similar between those patients completing the study vs. those who were lost to follow-up. Unfortunately, the possibility of differential attrition by unmeasured factors can never be excluded. Sensitivity analyses can assess whether extreme assumptions about patients lost-to-follow-up will affect study findings (e.g. assume that attrition is associated with 0% or 100% outcome rate).⁸ Statistical procedures have been proposed to adjust for attrition, although these require assumptions that often are unverifiable.⁶ The most obvious and preferred approach is to minimize attrition in the first place.

Challenges of Minimizing Attrition in Multi-site Emergency Care Research

Minimizing attrition can be challenging even under ideal research conditions, including the presence of adequate funding, experienced study personnel, and specialized research infrastructure. Attrition rates of 15–20% are commonly reported in longitudinal studies.^9–12 Several authors have identified factors predictive of attrition in specialized research settings. For example, the Baltimore Longitudinal Study on Aging found older age, lower education, and long distances from the study center were strongly associated with attrition.¹³ In addition, increased time to follow-up of study participants will reduce follow-up rates.^14,15 Several small studies cited patient-centered research staff and establishing rapport with trust as significant in overcoming some of these barriers to retention.^16,17

Conducting research in busy emergency department (ED) settings introduces additional challenges, including: lack of pre-existing care relationship with ED providers or hospitals may diminish engagement with the research process; acute illness and/or cognitive impairment may limit understanding of research protocols such as follow-up procedures; and patient populations difficult to reach after hospital discharge, such as those without stable housing or phone contact, language barriers and international travelers.^{5, 18–20}

Two prior articles have reviewed potential approaches to reducing attrition for emergency care research.^5,6 Table 1 describes general barriers and strategies to overcome such challenges.⁵ The specific approach must be tailored for each specific study, as the duration and intensity of follow-up procedures will vary greatly. For example, expected attrition will be lower for short-term outcome ascertainment by phone follow-up compared to a study requiring prolonged serial, in-person visits for formalized functional status testing.

Table 1.

Potential Attrition and Retention Problems and Solutions⁵

Potential Problem	Potential Solution	Pro	Con
Failed contact process	Point of enrollment contact confirmation	Real-time confirmation	Requires available technology e.g. cellphone; privacy; may limit economic diversity
	Intermittent, staged, post-enrollment contact (i.e. email/text; letter with home visit)	Improves awareness, reminders, updated contact information, home visit data; offer incentive for change of information notification	Time and cost for follow-up staffing, requires IRB approval for all post-enrollment contact
Mobile population	Phone follow-up	Alleviates distance /transport concerns	Lack of face-to-face follow-up
Lack of proximity to follow-up location	Self-reported follow- up	Obtains follow-up information	Reliability of self- report
	Central Phone Bank	Obtains follow-up information, standardized approach, decrease attrition	Time and cost for follow-up staffing
Lack of routine follow-up care	EHR-linked follow-up alerts	Obtains follow-up information	EHR interoperability and alert build
Limited funding for local follow-up	Offer no-cost, long term follow-up	Obtains follow-up information	Costly
Long duration of longitudinal follow- up	Shorten follow-up period	Decrease attrition	Limited to short term endpoints
Lack of objective endpoints	Use hard endpoints (e.g. death)	Follow-up with Death registries (SSDI) Identify clinical or disease specific objective endpoints	May not align with study outcome focus Limited consensus on objective endpoints

EHR (Electronic Health Record); SSDI (Social Security Death Index); IRB (Institutional Review Board)

Woolard et.al. previously described an approach to minimize attrition for a single site, randomized trial of a behavioral intervention in injured drinkers.¹⁸ Using a systematic approach to obtaining personal information, scheduling follow-up interviews, patient tracking procedures, and solving problem cases, they achieved attrition rates of 8% and 17%, respectively, at three month and one-year, in-person, follow-up interviews. Additional studies, including several from PECARN (Pediatric Emergency Care Applied Research Network), have achieved very low attrition rates through a rigorous pre-defined systematic approach, a waiver of informed consent when appropriate, low-intensity and communication-based follow-up processes.^19–21

Multi-site emergency care research introduces additional challenges to minimizing attrition. Research staff composition and approach to implementing study procedures will vary by site. Enrolling sites that dominate local health delivery may have an easier time tracking outcomes (e.g. multiple within-system hospitals linked by a common electronic medical records system) than sites in fragmented markets. Sample sizes are likely to be larger than for single site studies, magnifying the logistical challenges of follow-up. As research sponsors such as the NIH increase emphasis on supporting low-cost, pragmatic multi-site studies (i.e. those that do not require high-intensity follow-up procedures such as those described by Mello et. al. and Woolard et.al.), there is an important need to develop and describe procedures to minimize attrition for short-term outcomes identified through phone contact and chart review.^19–22

Case Example: The Syncope Risk Stratification (SRS) Study

We are the Steering Committee for an 11-site, investigator initiated, prospective observational study (NHLBI NCT01802398) called the Syncope Risk Stratification (SRS) study.²³ The goal of this study is to develop high value ED management algorithms for patients aged ≥60 years who present with syncope or near-syncope. The primary outcome is combined 30-day all cause death, cardiac arrhythmia, or other cardiac events (e.g. myocardial infarction). Eligible patients or their legally authorized representative (LAR) provide informed consent to participate in the study. Follow-up consists of direct phone contact with the patient or a LAR, supplemented by medical chart review. The institutional review boards (IRBs) of all participating sites approved this study.

The study coordinating center (which is also an enrollment site) is based in Portland, Oregon. The 11 enrollment sites are dispersed across the United States. As of May 1, 2016, we have enrolled 3,144 patients who are eligible for 30-day follow-up (i.e. we do not consider patients were enrolled in the prior 30 days). Our current attrition rate, defined as inability to achieve direct phone contact with patient or LAR, is 2.6%. Patients who dropped out were younger and had higher rates of four comorbidies (p<0.05) compared to those who completed follow-up. It is unclear whether these higher rates of comorbidities directly impacted attrition or whether these reflect access to care limitations, delays with reporting in the Social Security Death Index Master File (SSDI), or location changes due to need for long term care considerations.²³ However, for all patients without direct follow-up phone contact, we reviewed all available medical records at the enrolling site and performed a mortality query using the SSDI. The review of site specific obituary listings during the study period was not performed. There were otherwise no statistical differences on 15 other baseline variables (Table 2). Attrition rates were not statistically different between the enrollment sites.

Table 2.

Patient Characteristics for Completed Study and Lost to Follow-up

Finding	Completed Study (n=3062)	Lost to Follow-Up (n=82)	95% CI for difference of proportion
Age, mean (SD)	73 (9.0)	71 (8.7)
Gender
Male	1567 (51.2%)	44 (53.7%)	−2.5% (−13.0% – 8.4%)
Race
White or Caucasian (referent)	2556 (83.6%)	60 (73.2%)
Black or African American	393 (12.9%)	17 (20.7%)	−7.8% (0.8% – 19.3%)
Other	113 (3.7%)	5 (6.1%)	−2.4% (−1.0% – 12.6%)
Past Medical History
Baseline cognitive impairment or dementia	241 (7.9%)	4 (4.9%)	3.0%(−4.1% –6.1%)
Past stroke or transient ischemic attack	396 (13.0%)	10 (12.4%)	0.6%(−8.2% – 6.3%)
Congenital heart disease	36 (1.2%)	3 (3.7%)	−2.5%(−9.0% – 0.0%)
Congestive heart failure	375 (12.3%)	18 (22.0%)	−9.7%(−1.9% – −2.0%)
Ejection fraction < 40%a	104 (3.4%)	6 (7.3%)	−3.9%(−1.2% – 0.0%)
Peripheral vascular disease	168 (5.5%)	9 (11.0%)	−5.5%(−14.1%– 0.0%)
Implanted permanent pacemaker	241 (7.9%)	6 (7.3%)	0.6%(−7.2%–4.6%)
Implanted defibrillator	143 (4.7%)	2 (2.4%)	2.4%(−3.8%–4.2%)
Coronary artery diseaseb	848 (27.9%)	28 (34.1%)	−6.2%(−17.3%–3.0%)
Structural heart diseasec	123 (4.1%)	5 (6.1%)	−2.0%(−9.5%–1.5%)
Arrhythmiad	696 (22.9%)	13 (15.9%)	7.0%(−2.6%–13.4%)
Seizure disorder	59 (1.9%)	2 (2.4%)	−0.5%(−6.6%–1.3%)
Diabetes requiring medication	730 (24.0%)	28 (34.1%)	−10.1%(−21.2–0.8%)
Hypertension requiring medication	1998 (65.8%)	60 (73.2%)	−7.4%(−16.5%–2.7%)
Chronic renal insufficiencye	333 (11%)	15 (18.3%)	−7.3% (−17.2% – 0.4%)
Cancer requiring current active treatment	197 (6.5%)	6 (7.4%)	−0.9% (−8.7% – 3.1%)
ECG Interpretation
Normal	1513 (50.1%)	33 (41.3%)	8.8% (−1.8% – 19.3%)

^aBased on most recent testing within 1 year

^bIncludes angina, myocardial infarction, positive stress test, history of coronary artery bypass grafting, percutaneous transluminal coronary angioplasty

^cAortic stenosis, pulmonary hypertension, cardiomyopathy, valvular heart disease, valve disease, idiopathic hypertrophic subaortic stenosis

^dVentricular arrhythmia/sudden death, supraventricular tachycardia (paroxysmal atrial tachycardia, paroxysmal supraventricular tachycardia, atrial fibrillation, atrial flutter), sick sinus

^eCreatinine 1.5 mg/dL for at least 3 months

We describe seven features of our approach to minimize attrition in this multi-site study: 1. Patient selection; 2. Baseline contact data collection; 3. Patient incentives; 4. Patient tracking; 5. Central phone bank; 6. Local enrollment site assistance; and 7. Continuous performance monitoring.

Patient Selection

We a priori identified patients that were unlikely to complete phone follow-up either due to patient or study team factors. We excluded patients who were homeless or without a stable address, did not have stable access to a telephone (either landline or mobile), lived out of country or planning to move out of country within 30 days, or were incarcerated. We further excluded patients if both they and their LAR did not speak either English or Spanish as their primary language, limited by our study staff’s capacity to perform follow-up interviews in other languages.^5,6 Exclusion criteria reduce generalizability; however, we felt there was little chance such patients could complete phone follow-up. We opted to maximize internal validity at the expense of generalizability. However, this balance between validity and generalizability is one that each investigator must thoughtfully consider in the context of each study. Additionally, while not used in the SRS Study, implementation of a patient selection washout period can be considered. The goals of the study will dictate the washout period duration but the effects on the results and generalizability should also be considered.²⁴

Baseline Contact Information

Study staff obtained extensive contact information after an eligible patient provided informed consent to participate (Table 3). This included up to three personal phone numbers, an email address, whether personal address was a private residence or facility and optimal time of week and day for phone follow-up. If a patient provided a cell phone number, study staff were instructed to validate that information by calling the number while the patient was in the ED. Additional information to facilitate the follow-up call included patient’s preferred name and primary language. Finally, we collected contact information on up to two other people, who did not live with the patient, who would be able to locate the patient if primary attempts to contact the patient were unsuccessful. Though tempting to use ED registration information to save time and effort, obtaining and validating multiple contact options has significant benefits when following up on patients.⁵

Table 3.

Essential Patient Contact Information

Patient Incentives

Several reports support the common sense notion that patient incentives reduce attrition.^25–26 The use of incentives can pose ethical issues. For example, large monetary payments may induce acceptance of greater risk or result in disproportionate participation by economically challenged patients. On the other hand, incentives compensate participants for the value of their time and for any risk or discomfort involved in research participation. The predetermined patient incentives for the SRS study were reviewed by the IRBs and viewed to be ethically acceptable. At time of enrollment, patients were informed about a small gift card incentive for completing the 30-day phone interview.

The Coordinating Center institution mails gift cards of a small IRB approved reimbursement (<$25) to a national and ubiquitous retail store chain within four to six weeks after completion of the 30-day phone interview. The amount was determined based on previous experience for similar phone follow-up and time-based considerations. This retail store chain has a batch gift card program, which allows the Coordinating Center to efficiently purchase gift cards in bulk.

Patient Tracking

Tracking 30-day call windows for over 3,000 patients presents a formidable logistical challenge. Specialized research data management applications may simplify and automate this task. We use the Research Electronic Data Capture (REDCap) research management system that has been widely adopted by academic medical centers, including the SRS Coordinating Center.²⁷ Sites enter all information from screening and enrollment forms, including date of enrollment, within 48 hours. REDCap has a specific feature that tracks patients for follow-up activities within a pre-specified time interval after data of enrollment (Figure 1). Research staff at the SRS Coordinating Center has programmed REDCap to trigger a call list at 30 days. This call list remains open until the predefined last day for call – a 12 day window for the SRS study. While this tracking can take many different forms, it is critical to ensure proper follow-up occurs and that study leadership can quickly determine the status of follow-up. An automated system reduces human errors.

Figure 1.

Calendar Patient Tracking in RedCAP

Centralized Phone Bank and Procedures

In order to maximize uniformity and control over the call back process, we created a centralized phone bank managed by the SRS Coordinating Center. An automated system is not sufficient for accountability; the weekly assessment of the patient tracking by one Central Phone Bank project leader is required. The SRS Coordinating Center recruits and manages four to seven volunteer callers at any given time. Volunteers are typically post-baccalaureate graduates intending to apply for graduate training in health-related fields, or foreign medical graduates intending to apply for domestic medical residency training programs. Features of the centralized call center include a phone follow-up protocol with scripting (Appendix 1), standardized caller training including development of a manual of procedures, tips for common challenges prior to actual patient encounters, a predefined minimum of proctored patient interviews with an experienced caller, shadowed training shifts with an experienced caller, phone follow-up checklist created to minimize omissions during each call shift and staggered shift coverage to honor patient preferred follow-up days and times across multiple time zones. It is important to recognize that this system of volunteers may not be appropriate or may need to be modified for studies with substance abuse, mental health or other disorders with higher risk for potential call associated crises.^18,19

The Coordinating Center callers use the REDCap tracking function to identify patients who are within the window for 30-day phone calls. Callers will make up to 12 phone calls within a 12-day call window before resorting to alternative strategies discussed below. Callers will also attempt to contact patients’ alternative contacts. The 12-day phone window is often extended in uncommon circumstances (e.g. multiple attempts at disconnected phone number will extend the call window once the coordinating center receives a working phone number, patients travelling during holiday season, etc.). All calls placed are logged and tracked to minimize unnecessary calls and maximize efficiency. Data collected from the scripted protocol, including death (ascertainment by LAR), hospitalization, and ED visit after the index enrollment visit, are initially collected on a data form prior to being entered in real-time into REDCap. For patient reported hospitalizations and ED visits, callers collect the name, city, and state of the hospitals. Patients occasionally gave us inaccurate information about hospital name; therefore, callers verify hospital name information in real time by performing an internet search.

We have developed separate processes to independently verify patient reported events (e.g. “I was told that I have a heart problem,” when chart review of health encounter reveals that no cardiac conditions were identified). All subsequent visits to an enrolling site are verified by chart review performed by local study staff. To address visits to other facilities after the baseline enrollment date, study staff at local enrolling sites have patients sign an “omni” medical chart release form at the time of enrollment. This signed form allows Coordinating Center staff to request medical chart information from any hospital the subject subsequently visits within 30 days. The importance of this form was emphasized during the consent process given the inherent challenges with our patient population and ED location. This approach has been approved by all enrollment site IRBs and was justified as being crucial to capturing relevant clinical and health service utilization outcomes after the index ED visit. Coordinating site staff obtain and review non-enrollment site chart data.

Local Enrollment Site Assistance

The Coordinating Center staff work closely with local enrollment site research staff for patients who cannot be contacted within the 12 day call back window. We have found this collaborative approach to be highly effective in further reducing attrition. Occasionally a trivial and easily correctable error, such as a REDCap data entry error of a contact phone number, will be identified. Also, some patients are reluctant to answer a call from an out-of-state phone number. To address this at the time of enrollment, local study staff give patients a reminder card that they will be contacted in 30-days by an Oregon based phone caller. The specific Coordinating Center phone line number is included on the card. Finally, we discovered some patients are more willing to speak with local research staff they have previously met. We have avoided several cases of potential attrition by having local enrollment site staff complete the 30-day follow-up. As discussed earlier, real-time confirmation of phone numbers is also strongly encouraged at the site during baseline enrollment.

Continuous Performance Monitoring

We review several metrics related to 30-day follow-up on weekly phone conference meetings with project managers from all 11 sites. These include number of enrolled patients, patients within call window, patients at risk of lost-to-follow-up, patients lost-to-follow-up and monitoring for process adherence and performance across sites (Figure 2). Continuous performance monitoring allows the Coordinating Center to rapidly identify problems, their reasons and solutions. This also allows for ongoing effectiveness assessment of process controls to mitigate attrition and identify process failures that may negatively impact retention. Examples of some of the challenges rapidly identified and corrected include: turnover of Coordinating Center callers and local site research staff; delayed entry of enrollment data (REDCap tracking requires baseline information including date of enrollment); data entry errors of patient phone numbers; and patient reluctance to speak with non-local callers.

Figure 2.

Continuous Performance Monitoring of Follow-Up

Limitations

The various approaches to minimize attrition that have been described and employed in the SRS study will have variable impact depending on the nature of the research study. The small, lost to follow-up group in this study, appeared to have a slightly greater burden of disease which may reflect access to care limitations, delays with reporting in the SSDI, or location changes due to need for long term care considerations. Although a small group, further future study refinements should be considered to better ascertain why this may be the case.

Summary

Attrition is a major potential threat to study validity and power. Emergency care research is shifting towards enrollment through multi-site networks, but there have been limited descriptions of how to minimize attrition for these kinds of studies. Using the experience of the multi-site emergency care Syncope Risk Stratification study, we describe seven features that can be adopted to minimize attrition rates. Use of these features has resulted in a very low attrition rate in the SRS study. Appropriate study design and consideration of methods to reduce attrition will be time well spent.

Appendix 1: Sample Follow-up Scripting

Hello, Mr. /Mrs. _______________, my name is ___________. I work on a research project about the best way to take care of patients who fainted. You previously agreed to participate in our survey when you were seen on __ __/__ __/__ __ (date of enrollment) at the _______ Emergency Department. Please know that it is your right to decline to continue participating in this phone survey for any reason, at any time.

As you may recall, we wish to make sure that patients with an episode of fainting have not developed any dangerous medical problems. You previously agreed to complete a phone survey at 1-month after you were initially seen and gave us permission to request necessary medical records. We will ask you questions about your health and health care since your initial emergency department evaluation for your symptoms. Your participation today will require approximately ten minutes of your time. We will send you a giftcard to thank you for your time upon completion of this survey.