Lessons learned from two randomized controlled trials: CITIES and STOP-DKD

Abstract

Background

Even well-designed, theoretically driven clinical trials can fall short of achieving the desired clinical outcomes. Our research team had an opportunity to conduct two randomized controlled trials that were enrolling patients in parallel. While both studies were targeting chronic disease management among patients with multiple comorbid conditions, the patient population and settings varied. The studies were the Cardiovascular Intervention Improvement Telemedicine Study (CITIES) and Simultaneous Risk Factor Control Using Telehealth to slow Progression of Diabetic Kidney Disease (STOP-DKD) studies. Both studies had null findings.

Objectives

Our goal is to discuss common design considerations across CITIES and STOP-DKD and potential implications for the design of future randomized controlled trials.

Methods

These were two 1:1 randomized controlled trials with attention control groups that recruited patients from various clinical practices in the Research Triangle area of North Carolina.

Conclusions

We make three recommendations for future studies. First, we assert that it is important to allow for piloting the enrollment process to ensure that it is possible to identify and recruit a patient population that is well aligned with the clinical outcomes of the intervention. Second, analysis plans should be more targeted in their approach and should consider heterogeneity of treatment effects. Third, in order to support the transition of evidence generated from randomized controlled trials into clinical practice, it is important to consider even early stage randomized controlled trials through an implementation science lens.

Trial registration

Simultaneous Risk Factor Control Using Telehealth to slow Progression of Diabetic Kidney Disease (STOP-DKD) NCT01829256; Cardiovascular Intervention Improvement Telemedicine Study NCT01142908.

1. Introduction

There are known disruptions in the research pipeline. Approximately half of research has avoidable research design flaws, is unusable, incompletely reported or both, and/or research is never published [1,2]. Regarding the suboptimal reporting of research findings, investigators may not submit studies for publication when the results are inconsistent with their hypotheses [3]. Additionally, journals may be biased in their interest in publishing null trials because of a perceived lower societal attention attributed to studies with null results [4]. While a recent study demonstrated no difference in the post-publication metrics of randomized controlled trials depending on the direction of their findings [5], publication bias for null studies remains a problem in scientific reporting. Even when null studies are published, they are often viewed in isolation and not referenced or discussed in consideration of the full body of research in a given area. We assert that another form of research waste is not taking time to reflect on lessons learned. It is important for the research community to reflect on null studies, consider commonalities and implications for future research.

Even well-designed, theoretically driven clinical trials can fall short of achieving the desired clinical outcomes. Considering and learning from these experiences can be helpful to the research enterprise as a whole. Our research team had an opportunity to conduct two randomized controlled trials that were enrolling patients in parallel (Table 1). While both studies were targeting chronic disease management among patients with multiple comorbid conditions, the patient population and settings varied. The studies were the Cardiovascular Intervention Improvement Telemedicine Study (CITIES, NCT01142908) [[6], [7], [8], [9]] and Simultaneous Risk Factor Control Using Telehealth to slow Progression of Diabetic Kidney Disease (STOP-DKD, NCT01829256) studies [[10], [11], [12]]. CITIES focused on a patient population with cardiovascular disease risk factors including hypertension, dyslipidemia, or type II diabetes mellitus. Patients enrolled in the CITIES study received theoretically driven chronic disease management support, including blood pressure self-monitoring and medication management, provided over the telephone by a clinical pharmacist over the course of 12 months. The second randomized controlled trials, STOP-DKD, intended to focus on a population of patients with diabetic kidney disease. While the specific content of the intervention was varied, patients enrolled in STOP-DKD also received theoretically driven chronic disease management support, including blood pressure self-monitoring and medication management, provided over the telephone by a clinical pharmacist over the course of 36 months. Our goal in this paper is to discuss common design considerations across the two studies and potential implications for the design of future randomized controlled trials. Engaging both project managers and investigators, we arrived at these lessons learned based on a debriefing session among project leadership teams (see Table 2).

Table 1.

Overview of study characteristics.

	Basic Study Design	Patient Population	Intervention Delivery	Intervention Content	Setting
CITIES	Two-arm RCT of intervention vs. usual care	Adults with hypertension, dyslipidemia, and/or type II diabetes mellitus (n = 428)	12 monthly telephone calls delivered by a clinical pharmacist	Topics in chronic disease self- management; medication management provided by research pharmacists	Primary care clinics affiliated with a VA medical center in North Carolina
STOP-DKD	Two-arm RCT of intervention vs. usual care	Adults with diabetic kidney disease (n = 281)	36 monthly telephone calls delivered by a clinical pharmacist	Topics in chronic disease self- management; recommendations for medication management provided by research pharmacists totreating providers	Primary care clinics affiliated with an academic medical center in NorthCarolina

Table 2.

Key lessons learned.

1. It is important to allow for piloting the enrollment process to ensure that it is possible to identify and recruit a patient population that is well aligned with the clinical outcomes of the intervention.

2. Analysis plans should be more targeted in their approach and should consider heterogeneity of treatment effects.

3. In order to support the transition of evidence generated from RCTs into clinical practice, it is important to consider even early stage RCTs through an implementation science lens.

2. Methods/design

Key Similarities and Differences in the Studies. The CITIES and STOP-DKD studies had several key similarities and differences. Key similarities included their focus on complex patients with multiple comorbid conditions and use of electronic health record data to identify these patients. While the patient populations and enrollment criteria, and electronic health records were different, both studies began patient identification by obtaining an initial sample of patients from the electronic health record based on inclusion/exclusion criteria. Then within each study, research staff queried the electronic health record for patients meeting clinical criteria. These criteria included diagnoses (e.g., hypertension diagnosis), and well as whether clinical measures of these conditions were in control (e.g., recent blood pressure in excess of 140/90 mm Hg in a recent outpatient office visit). While the specific staff were different, both studies relied on doctoral-level clinical pharmacists (i.e., PharmDs) as interventionists and in both studies these pharmacists delivered scripted content to study participants over the phone. In the event that there was staff turnover, both studies ultimately had two pharmacist interventionists. Another similarity was that both interventions addressed medication management and adherence, although they did so in slightly different ways. Specifically, in CITIES the pharmacists could make medication dose changes, or add or remove medications in real time while communicating with a patient. In STOP-DKD, the pharmacists did not make changes in real time. Instead, they worked directly with the primary care providers for medication changes. Content that was delivered focused heavily on diabetes, hypertension, and diabetic kidney disease education. Each call also included medication management, addressed side-effects, and the collection of home blood pressure values which guided potential medication changes as needed.

Key differences in the study design included differences in the patient population, the recruitment setting, the scope of practice of the clinical pharmacists, and differences in the primary endpoint. CITIES focused on those with any cardiovascular risk factor (hypertension, dyslipidemia, or type II diabetes mellitus) and STOP-DKD focused on patients who had already been diagnosed with diabetic kidney disease or who were at risk of developing diabetic kidney disease. While the two studies recruited patients from the same community in North Carolina, CITIES recruited patients from primary care clinics affiliated with a Veterans Health Administration (VA) medical center, whereas STOP-DKD enrolled patients receiving care at a neighboring private academic medical center. Reflective of these two health care systems, the demographic characteristics of study participants was different. The CITIES study sample was predominately white (47%), partnered (57%), men (85%) with a median age of 62 years at diagnosis. The STOP-DKD study sample was predominately African American (55%), partnered (59%), male (52%) with a median age of 63 years at enrollment.

The specific activities of the clinical pharmacists varied between the studies. Clinical pharmacists practicing in the VA fall under federal policies and, compared with many states, have an expanded scope of practice. As a result, clinical pharmacists in the CITIES study were directly able to make prescribing and medication dosing changes in real time during telephone interactions with patients. In contrast in the STOP-DKD study, pharmacists could make written recommendations about medication changes to a patient's treating clinical providers. A clinical provider could review the pharmacist's recommendations in the electronic health record and then choose whether or not to follow the pharmacist's recommendation to make a medication change.

Another difference with the primary outcome measures was that CITIES evaluated Framingham cardiovascular disease risk as its primary endpoint [6,13]. The primary outcome of STOP DKD was change in estimated glomerular filtration rate. This is an important distinction as the data points that comprise these outcome measures is different. However, both outcomes for both studies relied on changes in systolic blood pressure.

Common Challenges. An important note about these studies is that while their design could be improved in hindsight, neither suffered from common design flaws. Both studies were adequately powered and hit patient recruitment targets. Both studies were randomized, single-blinded, and had robust control groups. Both studies had a strong theoretical underpinning, scripted intervention text, and fidelity monitoring. However, both study designs could have been improved. The research teams for both projects consider the projects to have been too intensive, particularly STOP-DKD which relied on monthly phone calls over 36 months. This intensity resulted in patient fatigue and some frustration (e.g., complaints of redundancy in content, etc.). It is also not an intervention that would be practical for many health care systems to implement.

The common problems experienced in CITIES and STOP-DKD require a deeper consideration. There were three common problems: 1) challenges identifying the target patient population; 2) differences in clinical pharmacists’ practice patterns; and 3) changes in the “usual care arm” for medication reconciliation programs. While these three problems were anticipated before the trials began, as the studies were executed it became apparent that the ramifications were more significant than was expected.

Both CITIES and STOP-DKD had difficulty finding the “right” patients. By “right” patients, we mean patients who not only met technical eligibility criteria for the study, but also were in a position to benefit from the intervention. As previously discussed, both studies screened the electronic health record to identify patients with specific diagnoses whose recent clinical values were out of control. Once enrolled, clinical values were subsequently collected by trained research staff using a standardized protocol. Blood pressure is one example that was collected across both studies. Blood pressure was measured based on Joint National Committee guidelines including having research staff taking multiple blood pressure measurements in a quiet space and 5 min of rest [14]. Research staff would measure a participant's arm and use an appropriately sized blood pressure cuff and ensure that the participant sat with his or her feet flat and shoulder width apart. The cuff was also applied to bare skin (as opposed to over a shirt sleeve). The blood pressure monitors were frequently calibrated and assessments were conducted three times per session to obtain an accurate value. This is in stark contrast to the way that blood pressure is typically assessed in a clinical setting. In most clinical settings, blood pressure assessments are conducted in a less structured way. Consider a patient who has struggled to find a parking place in a large medical center, then rushed into their clinic appointment. Clinic staff, equally burdened by scheduling challenges, may quickly take a patient's blood pressure over their shirt while the patient is sitting on a tall exam chair, and while asking them about medication changes and symptoms in tandem.

Perhaps as a result of this scenario, in both the CITIES and STOP-DKD studies, the study-measured blood pressure assessments were considerably lower than the clinic-measured blood pressure values. Specifically, the overall averaged baseline blood pressure at the time of enrollment was 130/76 mmHg and 134/76 mmHg for CITIES and STOP-DKD, respectively. In short, the enrolled participants had much better disease control than expected based on values obtained from the respective electronic health records; thus, the studies’ target populations were not precisely what was intended. This made it more challenging for the interventions to move the needle on disease control and show a clinically meaningful improvement.

Another common challenge across the two studies was differences in practice patterns of the clinical pharmacist interventionists. Clinical pharmacists are highly trained and skilled practitioners with individual approaches to disease management and communication. In general, practice guidelines for pharmacists varied by clinical setting, but also by clinical context many physicians were not familiar or comfortable with working with a pharmacist. One indicator of this was that physicians did not respond to a significant amount of the pharmacists' recommendations [11].

Both studies were 1:1 randomized controlled trials with a usual care plus education control arm. While the studies were conducted in different health care systems, there was a improvements in usual care. In the context of the CITIES study, the VA medical center, where patients were recruited, rolled out a medication reconciliation program. Much like the intervention content, this program involved interaction with a clinical pharmacist over the telephone focused on consolidating a patient's medication and synchronizing medication refills. This rollout occurred during patient recruitment and was available to participants in both the intervention and control arms.

Because the medication reconciliation program was part of usual care, it was difficult to determine which participants had participated in the program. Similarly, in STOP-DKD, there was an increasing use of additive programs providing overlapping material to both those in the intervention and control group. This type of contamination was unavoidable and often reflects what may increasingly occur in healthcare systems. A potential unintended consequence of this cross contamination is increasing burden on patients given the potential of overwhelming patients with duplicative communications. It will be increasingly important to align and integrate external programs with those that are occurring in the health care system.

3. Discussion

These two randomized controlled trials, CITIES and STOP-DKD, represented a significant investment in terms of research dollars, effort, and time. The studies occurred in real world clinics, which is critical for impacting the delivery of patient care and patient outcomes. However, it also comes with challenges. Based on our experiences with CITIES and STOP-DKD, we provide three specific recommendations for future cardiovascular disease self-management related randomized controlled trials.

First, we assert that it is important to allow for piloting the enrollment process to ensure that it is possible to identify and recruit a patient population that is well aligned with the clinical outcomes of the intervention. This process could involve conducting a research assessment to confirm clinical values (e.g., blood pressure measurements, etc.) prior to randomization. If, on average, the patient population is found to be in better control of their chronic conditions than expected, the trial design could allow for changes in the dose of the intervention to better use clinical resources. There is no need for a high-intensity, pharmacist-delivered intervention if patients are doing well and/or could be managed by a more economical skilled health professional such as a nurse.

Second, analysis plans should be more targeted in their approach and should consider heterogeneity of treatment effects. We know that there is no universally accepted approach to support all patients in achieving disease control outcomes. Statistical analyses should be planned to extend beyond simply addressing the research question of whether the intervention worked. Instead, we should ask for which specific sub-populations did the intervention work (e.g., people with certain demographic and/or clinical characteristics). This may require a different approach to sample size calculation and recruitment.

Third, in order to support the transition of evidence generated from randomized controlled trials into clinical practice, it is important to consider even early stage randomized controlled trials through an implementation science lens. By this we mean that using methods to understand the clinical context, including changes that occur in the context over the life course of the randomized controlled trial, is important. Randomized controlled trials may be more effective when a member of the research team, ideally the interventionist, is integrated into the health care system or clinic. Even when this is not possible, it is important to consider and articulate the core components of the intervention during the randomized controlled trial. If the trial is effective, this will inform a future understanding of where fidelity is important and what can be adapted to fit local context.