Design and implementation of pragmatic clinical trials using the electronic medical record and an adaptive design

Abstract

Objectives

To demonstrate the feasibility of pragmatic clinical trials comparing the effectiveness of treatments using the electronic medical record (EMR) and an adaptive assignment design.

Methods

We have designed and are implementing pragmatic trials at the point-of-care using custom-designed structured clinical documentation support and clinical decision support tools within our physician’s typical EMR workflow. We are applying a subgroup based adaptive design (SUBA) that enriches treatment assignments based on baseline characteristics and prior outcomes. SUBA uses information from a randomization phase (phase 1, equal randomization, 120 patients), to adaptively assign treatments to the remaining participants (at least 300 additional patients total) based on a Bayesian hierarchical model. Enrollment in phase 1 is underway in our neurology clinical practices for 2 separate trials using this method, for migraine and mild cognitive impairment (MCI).

Results

We are successfully collecting structured data, in the context of the providers’ clinical workflow, necessary to conduct our trials. We are currently enrolling patients in 2 point-of-care trials of non-inferior treatments. As of March 1, 2018, we have enrolled 36% of eligible patients into our migraine study and 63% of eligible patients into our MCI study. Enrollment is ongoing and validation of outcomes has begun.

Discussion

This proof of concept article demonstrates the feasibility of conducting pragmatic trials using the EMR and an adaptive design.

Conclusion

The demonstration of successful pragmatic clinical trials based on a customized EMR and adaptive design is an important next step in achieving personalized medicine and provides a framework for future studies of comparative effectiveness.

BACKGROUND AND SIGNIFICANCE

The importance of comparative effectiveness, understanding what treatment works for which patients under what circumstances, has gained increasing attention (1). The Federal Drug Administration (FDA) approval process requires a new treatment to be superior to placebo but not superior to established treatments. As a result, there are several approved treatments for common neurological disorders, but it is unknown which are superior in efficacy and tolerability, and for which subgroups of patients. It is possible, therefore, that some patients are receiving suboptimal care.

Furthermore, clinical trials, on which the prescribing recommendations are generally based, are often performed in highly selected patient populations using surrogate outcome measures over short time periods, and have been shown to generalize poorly to real-world patients and clinical practices (2–5). As such, there is interest in pragmatic trials that rigorously examine the effects of clinical decisions in the context that physicians are prescribing medications, not the “experimental” situation of a controlled trial. Inherently, randomized controlled trials are designed to explain a biological relationship between an intervention and the outcome, whereas pragmatic trials aim to guide decision-making at the point-of-care (6).

The success of pragmatic clinical trials depends on the active participation of providers, ability to capture data, and willingness of participants. Therefore, overcoming these potential barriers is essential to conduct this type of research. To increase provider participation, standardized data collection for the trial should be included in the normal workflow as much as possible, facilitated by electronic medical records (EMRs) (7), and for participants, pragmatic trials should not involve additional office visits outside of usual care. There are few tools available in EMRs to facilitate pragmatic trials in real-world patients; yet it is conceivable that discrete data captured in the EMR could be used to identify eligible subjects, to adaptively assign treatments, and to measure outcomes affordably and meaningfully at the point-of-care (8). While conceptually appealing, the operationalizing of such a system is challenging and requires specialized computing capacities that are still in their nascent phase of application (9, 10).

We have developed customized EMR tools to conduct pragmatic trials using a Bayesian subgroup based adaptive design (SUBA) (11), and to compare at the point-of-care the effectiveness of treatments. We describe, here, the implementation of this method in ongoing pragmatic trials for 2 neurological disorders, migraine, and mild cognitive impairment (MCI); our ability to capture necessary data; engage physicians and patients; and individualize medicine at the point-of-care (12).

MATERIALS AND METHODS

Protection of rights of human subjects

The protocols for these 2 trials have been approved by the NorthShore University Health System Institutional Review Board (IRB).

Data collection

We have developed a workflow to efficiently conduct adaptive pragmatic clinical trials in the context of the neurologists’ standard office documentation (either initial or follow-up visits). Data collection is done through our EMR (as previously described) (12, 13), which has been optimized to include structured clinical documentation support (SCDS) tools for 11 neurological conditions, capturing up to 1000 discrete data fields per office visit. Briefly, for each disorder, our SCDS “toolkits” capture baseline demographic and clinical characteristics, personal and family medical history and disease specific measures.

Study participants

Participants for our studies are drawn from the Department of Neurology at NorthShore University Health System (NorthShore), which includes 40 neurologists practicing at 4 hospitals and 8 outpatient sites in the north suburbs of Chicago, IL, USA. Participants are identified at the point of the care for an initial or follow-up neurology visit in the NorthShore network. Upon initiation of clinical documentation, a patient becomes a potential candidate for selection. Inclusion and exclusion criteria vary by condition and are determined by the EMR in real time to ascertain eligibility at the point of care. In general, we restrict to (1) individuals ≥18 years, (2) living in Cook or Lake County, IL, USA, and (3) who have not been previously treated with one of the potential assignment medications.

For patients meeting enrollment criteria (specific to each project, described below), a Best Practice Advisory (BPA) pops up when the neurologist accepts the progress note in NoteWriter. The BPA informs the neurologist that the patient is a candidate for participation in a pragmatic trial comparing 3 medications (specific medications indicated) using the EMR and SUBA (methodology discussed below), and a recommended treatment assignment will be provided (either randomly, for the first 120 assignments with outcomes; or adaptively using SUBA for the subsequent assignments, described below).

The BPA also provides the physician with specific verbiage to obtain verbal consent from the patient. The neurologist is asked to select “order” to indicate that the patient (or their proxy) verbally consents to enrollment in the study; or to select “do not order” if the neurologist or patient declines enrollment. If “do not order” is selected, the BPA requires selection of an acknowledge reason: ie, patient or family refused participation, physician defers medication, patient has a contraindication to either of the 3 drugs, physician prefers to optimize a current medication, physician prefers to use a medication other than the 3 compared, physician prefers to assign one of the 3 medications non-adaptively, BPA is in error (patient previously took or is taking one of the 3 compared medications), or other (specify). If the neurologist selects “order” in the BPA, a medication order is placed and pended for signature.

Patients are considered “enrolled” in the study upon clicking “order” in the BPA and then clicking “accept” (resolving) the BPA (intention to treat study design). Patients are considered “not enrolled” in the study upon clicking the “do not order” option in the BPA and then selecting an “acknowledge reason” from the drop down menu and then clicking “accept” (resolving) the BPA. Patients are excluded from the study if “order” and an “acknowledge reason” are selected and the BPA is accepted (illogical selections). The BPA can’t be accepted (resolved) unless either “order” or “do not order” are selected. A screen shot of the BPA for our migraine study is shown in Figure 1.

Figure 1.

Example of Best Practice Advisory (BPA) that opens for migraine patient when physician accepts progress note.

Treatment assignment

Treatment is assigned in 2 “phases” in this trial design. The trial begins with an equal randomization phase (phase 1), in which patients are randomly assigned one of the 3 study medications. Following this phase, the data from patients is used to start an adaptive randomization phase (phase 2).

Phase 1—equal randomization phase

The BPA stating a patient’s eligibility also provides a randomized medication assignment for the physician, as described above. This process appears instantaneous to the clinician, with no time delay in retrieving the medication recommendation. Patients are prescribed generic formulations to limit the burden of cost to the patient. At any time, dose adjustments can be made or the medications can be discontinued or adjunctive therapies can be prescribed according to routine clinical practices. Figure 2 illustrates how a request for a random treatment assignment is communicated from the EMR to the enterprise data warehouse (EDW, to create a back-up file), and communicated from the EDW to SUBA (to generate an input file), and then communicated from SUBA to the EDW (to create additional back-up files). Epic calls the randomization module via the Interconnect application. Interconnect provides a randomization allocation (medication assignment) for the patient in real-time to the physician. The randomization module will continue for new patients until 120 outcomes are collected. The EDW captures data as it moves through the EMR (specific details related to each trial described below).

Figure 2.

Example of data workflow implemented in subgroup based adaptive design (SUBA) trial using migraine information.

Phase 2—adaptive phase

SUBA will be used to guide the adaptive phase by processing information from the outcomes of the 120 randomly treated patients with outcomes. The results from phase 1 will be reviewed by the clinical and analytics team to ensure appropriateness of moving forward with adaptive randomization. SUBA applies a Bayesian random partition model to search for a suitable partition (clustering) of the patient space based on selected variables (11). SUBA can accommodate 3 independent variables, which are chosen a priori based on the specific project (described below). For each of the patients enrolled in phase 1, SUBA uses information on these 3 factors, their treatment assignment and their outcome. Based on the partition, SUBA calculates the posterior predictive probability that a future patient with specific variable values will respond to a particular treatment if the patient is assigned to the treatment. This treatment-specific posterior predictive probability is then used to randomize the patient. If the posterior predictive probability is larger for one treatment, the patient will have a larger randomization probability to be assigned to that treatment. In other words, patients are assigned adaptively to treatments based on predictive response. The posterior predictive probability for each future patient is continuously updated when new outcomes are observed from previous patients. This allows the trial to continue the learning until the end, potentially providing better benefits for patients in the trial by giving them a larger chance to be randomized to more desirable treatments.

If approved, SUBA will be used for at least 300 additional patients, as SUBA has shown desirable performance in computer-simulated trials with a sample size of 300 (11). Alternative sets of independent and dependent variables will be simulated using SUBA as well, but we plan to commit to our a priori selection of variables. The EDW will continue to input updated information for adaptively assigned patients to SUBA on a daily basis and the output file will be updated by SUBA each day (learning). The main statistical features of SUBA include the continuous learning of patient subgroups based on a random partition model and the adaptive allocation of patients to the best treatment arm based on posterior predictive probabilities (14).

Outcomes and follow-up

For the at least 300 adaptively assigned patients, we will capture outcomes data at initial, annual and interval visits (patient initiated visits between annual visits). Outcomes and follow-up are discussed below for each trial. We will perform an intent-to-treat analysis (ITT) consistent with the randomization.

Examples of implementation

We are currently in phase 1 of 2 pragmatic clinical trials, one for migraine and the other for MCI, each following the general structure discussed above. For each trial, we will randomize patients to achieve 120 outcomes and then adaptively assign at least 300 patients.

Migraine

Study population

Enrollment began on July 26, 2016 and is ongoing. As of March 1, 2018, 557 patients were eligible for inclusion in the study and 199 have been enrolled (36% enrollment rate, Figure 3A). As shown in Figure 3A, of the 358 not enrolled, 137 were due to the patient/family refusal. Of the remaining 221, the physician selected another reason for non-enrollment (details in Figure 3A). Specific enrollment criteria include (1) age 18 or older, (2) resident of Cook or Lake County, IL, USA, (3) initial visit type evaluated using the Neurology Headache SCDS toolkit, (4) diagnosis of migraine (without aura or with aura) as per the Impression smart form, (5) at least 1–3 headaches per month as per the Headache History smart form, (6) never took amitriptyline, propranolol, or topiramate as per the current Rx list or prior flow sheet, and (7) Migraine Severity and Disability Score (MIDAS) (15) and Center for Epidemiologic Studies Depression Scale (CES-D) (16) scores available as per documentation flow sheets. The CES-D and MIDAS are paper forms distributed to the patients at check in, and these score tests are self-administered by the patients in the waiting area prior to being roomed. The medical assistant rooms the patient, reviews the paper forms for missing items or illogical responses, and then enters the adjudicated patient responses into the EMR (as part of a pre-physician assessment). After the medical assistant has completed their assessments, the physician visits the patient and reviews prior medications information with the patient and also reconciles the information by reviewing available prescription records in the EMR (as part of a comprehensive assessment).

Figure 3.

A flow diagram of enrollment for (A) migraine subgroup based adaptive design (SUBA) trial and (B) MCI SUBA trial.

Treatment

Enrolled patients are randomized to one of 3 commonly used preventive agents, amitriptyline, propranolol, or topiramate.

Outcome

The primary outcome is survival free of either discontinuation (due to adverse effects or lack of efficacy) or survival free of adjunctive/alternative preventive medication(s) at 6 months (ie patients who complete 6 months of follow-up without discontinuing their medication or starting an adjunctive preventive medication are considered survival-free). Discontinuation is documented by an end date in the Medications list and with a discontinuation reason of either allergic response, alternate therapy, availability, cost of medication, ineffective, side effects/intolerance, or the subsequent prescription of an alternative preventive drug (ie any preventive drug, as listed in the Rx Med smart form). We follow patients though the EMR to determine the success (the drug was continued without an adjunctive added) or failure (the drug was stopped, and/or another preventive drug was prescribed) outcomes. The “Discontinuation” list is standard in Epic, and there are additional selections that we have determined are equivocal and do not provide outcome information. These include “Therapy Completed” and “Other”. Patients with these outcomes are, thus, censored and their outcomes considered non-informative. Their information does not contribute to SUBA. Outcome assessment is determined at 6 months. We chose 6 months given the following historical data (n = 868) from our practices. By 3 months, 100% of discontinuations occurred due to failure and after 6 months discontinuations were for unclear reasons. By 3–4 months half of the orders for an alternative drug were placed (and adjunctive therapy is one of our outcomes). If we extended follow-up to 18 months, we captured only a small number of additional orders. For currently enrolled patients, we are validating the use of the EMR to determine the outcome by calling individuals to verify their medication status, and reason for stopping, if the patient has discontinued use.

Covariates

The 3 independent predictors chosen were migraine subtype (aura or no aura), MIDAS score, and CES-D score. Based on historical data from our patients, migraine subtype and MIDAS were notably different between failed and successful drug assignments; while other factors were not. While some sub-scores of the migraine specific quality of life (MSQ) (17) also predicted failure, these scores were highly correlated with MIDAS (a single score). Additionally, principal components analysis revealed 2 distinct domain (MIDAS/MSQ and CES-D/GAD-7/Insomnia Severity Index; ISI (18)). Thus, we chose MIDAS and a factor from the second domain, CES-D, as depression had a higher prevalence in our patients (21%) than anxiety (9%) or insomnia (7%).

Mild cognitive impairment

Study population

Enrollment began on December 31, 2016 and is ongoing. As of March 1, 2018, 159 patients were eligible for inclusion in the study (enrollment is ongoing) and 100 enrolled in the study (63% enrollment rate, Figure 3B). Of the 59 patients not enrolled, 26 were due to patient/family refusal. The remaining 33 were due to the physician selected another reason (details in Figure 3B). Specific enrollment criteria include: (1) age 18 or older, (2) resident of Cook or Lake County, IL, USA, (3) initial visit type evaluated using Neurology Memory SCDS toolkit, (4) diagnosis of MCI (amnestic or other types) per the Impression smart form, (5) never took donepezil, rivastigmine, or memantine per the current Rx list or prior flow sheets, (6) Functional Activities Questionnaire (FAQ) score <9 as per flow sheet (19), and (7) Short Test of Mental Status (STMS) (20–22) and Geriatric Depression Score (GDS) (23) available as per documentation flow sheets. Restricting enrollment to FAQ <9 avoids enrolling individuals with decisional impairment. The FAQ and GDS paper forms distributed to the patients at check in, and these score tests are self-administered by the patients (or their proxy) in the waiting area prior to being roomed. The medical assistant rooms the patient, reviews the paper forms for missing items or illogical responses, and then enters the adjudicated patient responses into the EMR (as part of a pre-physician assessment). After the medical assistant has completed their assessments, the physician visits the patient and reviews prior medications information with the patient and also reconciles the information by reviewing available prescription records in the EMR (as part of a comprehensive assessment). The physician also administers the STMS and the patient responses are entered directly into the EMR.

Treatment

Enrolled patients are randomized to oral donepezil, oral rivastigmine, or oral memantine, all of which are commonly used to treat MCI in neurology clinical practice.

Outcome

The primary outcome of interest will be 1-year survival free of functional impairments (defined as FAQ <9). Specifically, patients are considered survival-free of functional impairments if at the 1-year follow-up their FAQ < 9. We consider success to be FAQ < 9 at 1-year, while failure is FAQ ≥9 at 12 months. FAQ is measured though NorthShore Connect (our EMR’s patient portal) 1-week prior to the first annual follow-up visit, typically 12 months. If not completed through the patient portal, the FAQ is administered at the annual visit. Persons who do not return for follow-up within 14 months of enrollment (less than a third do not return, annual follow-up rate is 72%) are telephoned by a Research Assistant to complete the FAQ as a scripted interview. If we are unable to reach them by telephone after 2 attempts, they will be censored and their outcomes will not contribute to SUBA.

Covariates

The 3 independent predictors are MCI type (amnestic vs other), STMS, and GDS. Using principal components analysis to identify sources of variation in FAQ scores in our previously collected data (n = 365), we identified 2 measures, STMS and FAQ that had first principal component loading > 0.40. We included MCI type as our third variable as it is unclear if nootropics will help patients with memory impairment (vs other MCIs) the most.

DISCUSSION

Pragmatic trials are receiving increasing attention as a method to conduct quality improvement and comparative effectiveness research (24). Much of their appeal draws from the need to address issues inherent in controlled studies including generalizability of findings and questions of relevance to clinical decision making (2). Herein, we describe the design for pragmatic, adaptive, clinical trials of comparative effectiveness, using data from our optimized EMR and our SCDS “toolkits”. This is a crucial aspect as “integrated comparative effectiveness” research in comparison to “segregated comparative effectiveness” research (eg clinical trials) is intended to be conducted with the minimal disruption to clinical workflow by clinicians who may or may not consider themselves to be engaged in research (24). As such, the EMR becomes indispensable as patients should also experience minimal disruption in their typical clinical care. Our substantial optimization of the EMR for the purposes of quality and comparative research effectiveness is novel and innovative. This point-of-care approach has been discussed as an efficient solution to previous designs of pragmatic trials and “a bridge at implementation” (25) and has been undertaken in a handful of studies (26, 27). We have gone one step further with this design by incorporating an adaptive treatment assignment using SUBA, which has been shown to require smaller sample sizes in simulation studies (11) and to increase the probability that patients will receive an effective treatment (28), thus improving health at the individual level during the course of the trial. Pragmatic point-of-care trials are challenging and require significant bioinformatics support, including a system to manage randomization assignment in real-time, and additional computing and statistical resources are needed for adaptive trials (10).

Demonstrating the feasibility of this approach is of utmost importance, as, there is a relative paucity of studies of pragmatic trials of comparative effectiveness compared with the amount of literature highlighting the theoretical relevance. There are several obstacles to address to conduct these studies, both from an operational and research standpoint. The burden of providers having to consent at the point-of-care has been noted as problematic in previous trials (29, 30). To address this, we have developed BPAs that pop-up if a patient is eligible based on the information the provider has entered in the EMR, thus eliminating the need for the physician to evaluate a patient’s eligibility; and a script for consent is provided when the BPA fires, resulting in little additional time required during the office visit. Additionally, because all medications are FDA approved and routinely prescribed for the given indication, we were approved by our IRB to consent patients verbally, reducing additional physician workload. Further, the importance of engaging the clinicians in the research objectives is imperative to ensure active participation (24). As previously discussed (12), the success of the development and use of the SCDS toolkits is partly due to early and frequent engagement of clinicians in each practice area. The success of engagement is evidenced by our enrollment rates (36% and 63%). Though these may seem low, we had very broad inclusion criteria to fully represent our patient population. Traditional clinical trials generally impose more strict inclusion criteria. Importantly, because follow-up is conducted through the EMR, patients are not burdened with visits to remain in the study.

Despite the innovation and success, there are some limitations. It is possible we will not be able to recruit 300 additional patients (phase 2). Given our recruitment timeline for the initial 120 patients, and the prevalence of the condition and current volume of patients at our hospital, that seems unlikely. Clearly, for more selective enrollment criteria or rare outcomes, this study design approach will be more challenging and may require extending trials to other sites. We have developed a Neurology Practice Based Research Network (12, 13). Partner sites implement our “toolkits” to facilitate standardized multicenter quality improvement and research. As such, we intend to extend these trials to other sites that will represent diverse patient populations. This will also represent a more heterogeneous mix of providers, again increasing the generalizability and translation of findings to clinical practice (7).

Misclassification of some variables is possible. For example, whether a patient has ever taken the study drug is determined through information in the EMR and self-report. It is possible that patients received care at a different hospital system and do not recall accurately their previous medications. Similarly, because we rely on the EMR for our outcomes in the migraine trial, some misclassification is possible if the EMR does not accurately reflect the patient’s medication usage at the end of the trial period (6 months). To validate the use of the EMR for medication outcomes, we are conducting a scripted telephone interview to inquire about discontinuation. As of March 1, 2018 we have contacted 126 patients and found that the sensitivity of the EMR is high for detecting successes (95%). However, the specificity is low (54%), with almost half the individuals reporting failure not having their outcome accurately reflected in the EMR. Given this, we are now conducting telephone interviews with all enrolled subjects in the migraine trial, and will use this as our source of outcome information. We are also working to establish a workflow to update a patient’s EMR medication list, so that the EMR and the patient’s care team have more accurate information as it relates to patients’ migraine treatment. There may still be misreporting by patients, however, any misclassification is likely to be non-differential and bias towards the null. To induce a spurious association, misclassification of medication use would need to be associated with both treatment assignment and treatment response, which seems unlikely. Going forward, we are also planning to implement measures to improve the accuracy of the medication record in the EMR. First, we will educate patients enrolled in our trials, on the importance of communicating migraine treatment medication changes to their care team. Second, for those enrolled in our trials, we will be sending monthly electronic messages via our secure patient portal reminding them to notify their care team of any changes in their migraine treatment. We will continue the phone interviews, as described, to compare the patient reports and EMR accuracy after this change to our trial workflow. In our telephone interview, we also found that some individuals (19%) never start the study drug and these were rarely documented in the EMR. Consistent with the intent-to-treat design, these individuals are considered failures. As above, it is unlikely that this would induce a spurious association and most likely would cause a bias towards to the null. In the MCI trial, the outcome (FAQ) is a validated instrument and captured in the course of routine clinical care. Additionally, our follow-up rates in the MCI patients for first annual visit is high (72%), suggesting little need for additional study related work outside of routine care.

We are also only currently able to input 3 vectors into SUBA. An enhanced model has been developed that will allow for more flexible sub-group assignment (14). However, we used a combination of literature review and empirical analysis of our populations to choose strong candidates to predict treatment response.

Because our study is un-blinded (physicians and patients know their treatment assignment at the time of randomization), there is the possibility of bias. However, we have chosen objective outcomes, and, in reality, pragmatic trials are aimed at comparing treatments under clinical conditions in which neither the doctor nor patient are blinded to treatment assignment. Even if blinded at the time of treatment assignment, physicians will learn the assignment when they write/sign the prescription, and blinding physicians may result in substantially decreased enrollment rates. Additionally, un-blinded assignment represents the real-life scenario under, which this clinical decision support would be operationalized in clinical practice. Physicians must have knowledge of the order they are prescribing, and can choose to not prescribe the assigned treatment if they believe it is not appropriate or the patient has a contraindication. To mitigate the potential of subtle biases due to un-blinding, all physicians were extensively involved in the design of the trial and agreed to use this method to guide treatment assignment. We further monitor enrollment rates and assess for evidence of selective enrollment according to medication assignment. Acknowledging these potential limitations, this pragmatic EMR-based adaptive assignment methodological approach remains a valuable step in progression towards providing precision medicine.

CONCLUSION

The proof-of-concept method presented here represents an innovative design for point-of-care clinical trials of comparative effectiveness, using adaptive randomization and an optimized EMR. This study design and our findings could represent important next steps towards providing precision medicine.