Recruitment experience for a pragmatic randomized controlled trial: Using EMR initiatives and minimizing research infrastructure

Abstract

Context

Modernized approaches to multisite randomized controlled trials (RCT) include the use of electronic medical records (EMR) for recruitment, remote data capture (RDC) for multisite data collection, and strategies to reduce the need for research infrastructure. These features facilitate the conduct of pragmatic trials, or trials conducted in “real life” settings.

Objective

We describe the recruitment experience of an RCT to evaluate a clinic-based intervention targeting urban youth with asthma.

Materials and Methods

Using encounter and prescription databases, a list of potentially-eligible patients was linked to the Epic appointment scheduling system. Patients were enrolled during a scheduled visit and then electronically randomized to a tailored versus generic online intervention.

Results and Discussion

1146 appointments for 580 eligible patients visiting 5 clinics were identified, of which 45.9% (266/580) were randomized to reach targeted enrollment (n=250). RDC facilitated multisite enrollment. Intervention content was further personalized through real- time entry of asthma medications prescribed at the clinic visit. EMR monitoring helped with recruitment trouble-shooting. Systemic challenges included a system-wide EMR transition and a system-wide reorganization of clinic staffing.

Conclusions

Modernized RCTs can accelerate translation of research findings. Electronic initiatives facilitated implementation of this RCT; however, adaptations to recruitment strategies resulted in a more “explanatory” framework. .

INTRODUCTION

Pragmatic trials are designed to evaluate the effectiveness of interventions under real-life routine practice conditions, and to inform clinical decisions and practice. Explanatory trials aim to test whether an intervention works under optimal conditions, seeking to understand how and why an intervention works in the ideal setting¹. One of the main tenets of the pragmatic trial is generalizability (external validity), or the extension of research findings to the population at large.² In comparison, the rigorous nature of explanatory trials seeks to maximize internal validity, or the accuracy of the research conducted. This includes achieving an adequate sample size for precise measurement of an intervention effect. ² Although both pragmatic and explanatory trials are important in the evaluation of healthcare interventions, pragmatic trials are thought to both facilitate and expedite translation of research findings into routine practice.^3,4

In 2008, the CONSORT and Pragmatic Trials in Healthcare (Practihc) groups created an extension to the CONSORT checklist for reporting of pragmatic trials.⁵ The checklist provides the basis for this manuscript. We also take this opportunity to use the PRECIS diagram developed by Thorpe et al, which illustrates how most trials consist of both pragmatic and explanatory features. We use this tool to summarize and display how our trial was positioned on the continuum at baseline, and how the position shifted as we progressed through the trial.³

In a previous publication, we described the design of a pragmatic randomized controlled trial (PRCT) to evaluate a behavioral intervention directed at urban teens with asthma in response to the NHLBI RFA-HL-12-019 (National Institute of Heath, National Heart, Lung and Blood Institute Request For Applications) entitled “Pilot Studies to Develop and Test Novel, Low-Cost Methods for the Conduct of Clinical Trials”. The intervention selected for this trial was the computer-tailored, online asthma management program known as Puff City. ^6,7 Asthma continues to be a major public health problem in the US with high economic and social costs. ^8,9 Low income ethnic communities are disproportionately affected by asthma as demonstrated by higher morbidity and mortality for these communities.^8,10 This is the first time Puff City has been evaluated as a clinical tool or one that is initiated in a clinical setting.

We incorporated components of our electronic health record as well as our Oracle Clinical data management system into the design of this trial, and included features such as a website for collection of patient reported outcomes, and remote data capture (RDC) to facilitate multisite enrollment and medical chart abstraction by clinical and research staff. These features were designed to reduce the need for development of additional research infrastructure to conduct the study, thereby reducing costs. We use the CONSORT checklist for reporting pragmatic trials as guidance in describing our recruitment experience for this trial. ⁵

METHODS

Patient identification

To identify potentially eligible patients, EMR encounter and prescription claims data bases (EMR back-end) were combined to apply criteria for persistent asthma from the Healthcare Effectiveness Data and Information Set (HEDIS) used to construct the denominator for the measure “Use of Appropriate Medications for People with Asthma.” ¹¹ A computer algorithm linked these patients with the centralized appointment scheduling system (EMR front-end) to retrieve upcoming clinic visits, and generated a listing of eligible patients with upcoming appointments using a 30 day window. The listing was refreshed weekly to include patients that had recently became eligible or were new to the health care system and may have been candidates for enrollment.

Electronic consent

To reduce clinic staff time needed to obtain informed consent in a busy clinic, we implemented an e-consent feature. Once an appointment was identified, a letter explaining the study was sent to the home of the potential participant. The letter contained a link to an online consent and assent form (e-consent) in English, Spanish, and Arabic. This allowed caregivers and youth to provide this information online, prior to the scheduled appointment. During the scheduled visit, clinic staff confirmed patient and parent/guardian consent/assent, or initiated the consent process.

Intervention Arm

Puff City is a web-based, asthma management tool that uses tailoring, defined as the “assessment and provision of feedback based on information that is known or hypothesized to be most relevant for each individual participant of a program.”⁴ Computer algorithms are used to assemble theory-driven feedback based on the user’s characteristics, beliefs, and attitudes, creating an extensive array of message permutations. Topics addressed in Puff City include controller medication adherence, keeping an inhaler nearby, and smoking reduction or cessation. Also included is information on trigger avoidance, instructions on using medication delivery devices and basic asthma physiology. Behavioral theories applied in Puff City include the Transtheoretical Model, the Health Belief Model ¹², and aspects of Motivational Interviewing (MI). “Scores” for asthma self-regulation are derived using the Asthma Self-Regulation Development Interview, ¹³ developed by Zimmerman et al., which describes sequential "phases" of asthma self-regulation including asthma symptom avoidance, asthma acceptance, asthma compliance, and asthma self-regulation. ¹³ Patients randomized to the treatment group received four online sessions designed to be no less than 1 week apart. A booster session at six months was created to sustain positive change and correct early stages of relapse.

Control Arm

Controls received four sessions of generic, online, asthma education. After login, control students were given access to links to four generic asthma websites. To regulate dosage, control teens received a “time expired” message after 30 minutes of browsing, which corresponded to the maximum time needed to complete a tailored session (intervention group). We selected control websites from recognized U.S. and Canadian organizations that had a history of providing evidence-based information on asthma management.¹⁴ Internet access was not required for participation. Upon enrollment, youth were given a packet containing information on how to access Puff City on a computer with internet access, a letter asking permission for students to access Puff City from a school computer, and information on computer resources in the area.

Randomization

After completing the baseline assessment, teens were able to access the first online session. Upon the completion, teens were automatically randomized to the intervention or control group.[30]

Patient enrollment

A centralized database with remote data capture (RDC) allowed clinical and research staff to enter teen enrollment information at the time of the index visit, as well as enter asthma medications prescribed at or within 24 hours after the visit. The latter enabled the intervention to provide patients with tailored information on asthma medications and medication delivery devices.

Follow-up

Four online educational sessions were followed by 6 and 12 month follow-up surveys for endpoint collection. For teens in the intervention group, the 6 month follow-up was immediately followed by a tailored booster session. Automatic email and text reminders were sent to patients to maximize compliance with the study protocol. Telephone reminders were used to motivate non-compliant patients.

Recruitment monitoring

The EMR created a valuable recruitment monitoring tool. In addition to number of enrolled, weekly monitoring reports showed total number of visits expected, “no-shows”, cancellations, appointments missed by the clinic or research staff (see Challenges), same-day appointments missed or enrolled, and refusals. This report was generated by clinical site, and by building upon programming already in place for the identification of eligible patients. This snapshot helped the research staff pinpoint problems and patterns potentially detrimental to the study (e.g., large number of refusals at one site).

Pragmatic features

Table 1 shows key differences between explanatory and pragmatic trials and the Puff City features corresponding to each domain. We provide a score corresponding to a location on the Explanatory-Pragmatic continuum we believe the feature resides, where 0 is most explanatory and 5 is most pragmatic. Figure 1a illustrates the pragmatic nature of our study using the PRECIS diagram presented by Thorpe et al, in 2009.³ The following are study details for each domain.

Table 1.

Key differences between trials with explanatory and pragmatic attitudes, adapted from [5] with permission of the BMJ Publishing Group Ltd.

	EXPLANATORY	PRAGMATIC	PUFF CITY	Initial Score (0–5)	PUFF CITY - MODIFICATIONS	Score post- modifications (0–5)
Question	Efficacy – can the intervention work?	Effectiveness – does the intervention work when used in normal practice?	Does Puff City work when initiated in primary care?	5		5
Setting	Well-resourced, “ideal” setting	Normal practice	HFHS Primary care clinics	5		5
Added infrastructure	Research infrastructure needed in terms of additional study visits	No additional research infrastructure needed	Minimize research infrastructure by recruiting during scheduled visit	5	Increased enrollment by research staff	3
Specialized staff	Trained research staff needed to consent, recruit, enroll	No additional research staff needed	Clinical staff is trained to consent, recruit, enroll	4	Trained research staff needed to consent, recruit, enroll	2.5
Participants	Highly selected. Poorly adherent or patients with conditions able to dilute effect are often excluded	Little or no selection beyond the clinical indication of interest	Inclusion criteria were age, asthma and appropriate consent/assent	4		4
Intervention	Strictly enforced and adherence is monitored closely	Applied flexibility as it would be in normal practice	Tech-based, tailored intervention	4		4
Follow-up	Study individuals followed with study- related visits and follow- up surveys	No formal follow-up of study individuals; use only admin databases for detection of outcomes	Study participants followed with study-related visits and follow-up surveys	1		1
Outcomes	Often short-term surrogates or process measures	Directly relevant to participants, funders, communities, and healthcare practitioners	ACT, functional status, quality of life relevant to patients and providers	5		5
Relevance to practice	Indirect – little effort made to match design of trial to decision making needs of those in usual setting in which intervention will be implemented	Direct – trial is designed to meet needs of those making decisions about treatment options in setting in which intervention will be implemented	Trial results can immediately be used in decision-making of patient and practitioner	4		4

Figure 1.

The Puff City intervention

Research Question

We assessed the ability to implement and evaluate Puff City in a clinical setting.

Setting

Pragmatic trials are conducted in a “real world” setting. The RCT to evaluate Puff City was conducted in HFHS primary care clinics.

Added Infrastructure

Trials are less pragmatic if they require the creation of new infrastructure specifically to conduct the evaluation. The more specialized the infrastructure for research needed, the less pragmatic the trial. To minimize the need for additional infrastructure, we elected to recruit eligible patients during a regularly scheduled clinic visit and to have clinic staff conduct enrollment and follow-up.

Specialized staff

Five HFHS Pediatric Clinics participated in the study. To minimize the need for specialized research staff, clinic staff were trained to recruit, consent and enroll patients. Clinic staff were also trained to make reminder calls for intervention sessions and follow-up surveys. All study activities were incentivized for the clinics. Payments were based on data accuracy and completeness as well as study milestones representing patient progression through the study.

Participants

Teens aged 13–19 years and visiting a primary care physician at one of the participating clinics (for any reason) were eligible for the study. Exclusion criteria included previous enrollment in an investigational asthma management /education trial, inability to use the program in English (Puff City is currently only available in English); or teen/caregiver inability to complete assent/consent, even with help.

Outcomes

The primary endpoint of the trial is asthma control test (ACT) at one year post randomization. Secondary endpoints included (S1) asthma hospitalizations, emergency department visits, or oral corticosteroid dispensing during the 12 month follow-up period, and (S2) self-report of functional status.

Relevance to practice

The delivery of asthma education in primary care is challenging. If effective, primary care physicians and specialists can provide this program to complement and supplement professional advice and consultation about asthma. Puff City content is based on recommendations from the Expert Panel III report.¹⁵

RESULTS

Recruitment

Figure 4 is a CONSORT Flow Diagram showing the flow of participants through each stage in recruitment. A total of 1146 appointments were identified for 580 youth who met criteria and had scheduled visits during the recruitment timeframe. A total of 7 patients (2.6%) made use of the e-consent/assent process. Refusals were similar to those that enrolled in terms of sex (p=0.41), mean age (p=0.72), race (p=0.23), ethnicity (p=0.91), and gaps in health insurance in the last month (p=0.56). In our study, 21.3% of refusals gave no reason for non-participation, while reasons most offered for refusal were “not interested” (47.9%), and “no time/too busy” (10.6%). Reasons such as “asthma under control” (2.1%) or “no internet access” (1.1%) were rarely given as a reason for not participating. The remaining reasons for refusal included physician discretion (5.3%), language barriers (2.1%), patient doesn’t have asthma (2.1%), or had to be categorized as “Other” (7.4%).

Figure 4.

CONSORT chart for Puff City pragmatic randomized trial

Figure 5 illustrates our progress toward a targeted enrollment of 250 teens. We were able to enroll 66.2% of those randomized (n=176) during a scheduled clinic visit. The remaining 33.8% (n=90) were enrolled at a subsequent clinic visit (n=58) or were enrolled at a “research” visit (n=32).

Figure 5.

Puff City pragmatic randomized trial targeted enrollment

Challenges

Hurdles to overcome during recruitment included a system-wide EMR transition, problems detecting same-day visits, clinic appointment no show/cancellation rates, and missed opportunities for recruitment.

EMR identification of appointments for potentially eligible patients

At HFHS, the transition from the previous application to EPIC application began 1/1/2013 which corresponded to our sixth month of recruitment (Figure 5). Alerted by an increase in missed opportunities for recruitment, programming staff investigated and found that our patient identification process was disconnected from EPIC resources being updated during the transition. This disconnect occurred over a period of two months.

Same-day appointments

In addition, “same-day’ appointments were frequently missed by our appointment retrieval programs. Unable to identify the problem, we resolved to dedicate staff time to manually conduct surveillance for same-day appointments every morning using the Epic appointment scheduling system. Approximately 32% of the 1146 appointments identified were same-day appointments identified in this manner.

Clinic staff recruitment

Clinic staff were incentivized to recruit and enroll patients for the study. Two months into study recruitment, a system-wide reorganization reduced clinic staff availability, and this method of enrollment was discontinued at all but one clinical site. Research staff were added to assist clinics with recruitment. (Figure 5)

No shows and cancellations

Of the 1146 appointments identified, 362 (31.6%) had a final disposition of no-show or canceled (data not shown). Of the 580 patients for whom we identified scheduled visits, 48 (8%) never appeared for a clinic visit, despite rescheduling of the appointment by the clinic or the patient several times over the recruitment period. Clinic no-show/cancellation rates averaged about 30% of visits throughout the study period, and this did not seem to vary significantly by season, although one clinic had rates consistently higher than others.

DISCUSSION

The purpose of RFA-HL-12-019 was to conduct RCTs more efficiently and at a lower cost. Applicants had to address a number of challenges, including strategies to minimize the creation of specialized infrastructure. This directly coincides with pragmatic trials which seek to conduct RCTs in real world settings. Strategies used in this study to lower costs without compromising internal validity included the use of EMRs for trial patient identification and the use of centralized appointment scheduling for patient recruitment, and enrollment. A combination of these resources were used to monitor recruitment progress. Another strategy to lower costs was to minimize visits designed solely for the study, by recruiting during a scheduled visit and enlisting the help of clinic staff in patient enrollment. We encountered difficulty, despite incentivizing study-related activities. Of the four trained and participating clinics, only one clinic was able to continue with staff enrollment of patients. This clinic provided 17% of our enrolled and randomized sample. In hindsight, a number of adjustments might have made clinic staff enrollment more feasible, including a streamlined recruitment protocol (staff had to login to two different systems to complete enrollment of a patient), and an effective strategy to ensure informed consent prior to the clinic appointment. Having limited staff and time to devote to research remains a problem for pragmatic approaches.

We provided potential participants the opportunity to e-consent at home or prior to the visit, however, uptake was low. Potential participants would not have been aware of the e-consent process without reading the letter that was sent to the home. This indicates that our study introductory letters (which were followed by a phone call) were only read by a small number of patients. Technically, e-consent seems feasible as a cost-savings or time-savings measure, however, effective strategies to reach potential participants prior to a clinic visit are needed.

Pragmatic and explanatory trials exist on a spectrum. We felt our study scored highest in areas of eligibility, intervention flexibility, and setting. As our study progressed, it was necessary to implement “explanatory” measures to reach targeted enrollment, thereby attempting to maximize precision in measurement of an intervention effect. This included the abandonment of clinic staff recruitment and follow-up. After these adaptations (for which we recalculated our scores in Table 1, our trial moved considerably toward the explanatory end of the spectrum (Figure 3b).

Figure 3.

Figure 3a. PRECIS-like diagrams with CONSORT and added domains representing original pragmatic study design adapted from [3] with permission of Elsevier

Figure 3b. PRECIS-like diagrams with CONSORT and added domains representing original pragmatic study design adapted from [3] with permission of Elsevier.

Previous clinic-based recruitment (without the EMR) strategies might have involved obtaining a listing of patients meeting eligibility criteria and soliciting patient visits (by phone or mail) to a research-based site, with an offer to reimburse participants for transportation. The strategies used in our study are pragmatic in that the study was conducted in a clinic setting. Conducting research in the “real world” provides information useful for future RCTs and acceleration of translation into practice.

CONCLUSION

We applied a pragmatic approach to the conduct of a RCT to evaluate an asthma intervention for adolescents. Pragmatic features included (1) using the EMR for patient identification, enrollment and study monitoring; (2) minimizing the need to establish specialized research infrastructure; and (3) conducting the trial in the clinic, (“real world”). Although these features may represent a low-cost alternative to traditional strategies for RCTs, the clinic setting presented challenges that shifted our trial toward the “explanatory” end of the spectrum. A pragmatic approach to conducting the RCT was feasible, however, we struggled to maintain the desired balance in features representing internal and external validity.

Figure 2.

Design of the Puff City Pragmatic Randomized Controlled Trial (PRCT)