Embedding clinical interventions into observational studies

Abstract

Novel approaches to observational studies and clinical trials could improve the cost-effectiveness and speed of translation of research. Hybrid designs that combine elements of clinical trials with observational registries or cohort studies should be considered as part of a long-term strategy to transform clinical trials and epidemiology, adapting to the opportunities of big data and the challenges of constrained budgets. Important considerations include study aims, timing, breadth and depth of the existing infrastructure that can be leveraged, participant burden, likely participation rate and available sample size in the cohort, required sample size for the trial, and investigator expertise. Community engagement and stakeholder (including study participants) support are essential for these efforts to succeed.

1. Introduction

The field of epidemiology has well-developed standards for the definition and conduct of many types of studies. This has resulted in a somewhat artificial dichotomy between clinical trials and observational studies in research, training and practice. Observational studies can include cohort studies and patient registries. Clinical trials may include randomized or non-randomized efficacy or effectiveness trials of drugs, procedures or behavioral interventions. Trials could target individuals within a cohort with specific phenotypes or attributes. Trials to consider could include implementation or policy trials at individual (patient), clinic level, or community levels. Hybrid designs can include embedding clinical trials into existing observational studies or designing joint observational and trial components together in a single unified design.

In planning new studies, investigators should consider from the beginning a framework that includes the potential opportunities for concurrent or future intervention research or opportunities to extend clinical trials with an observational component. The factors that favor hybrid vs. separate study designs should be articulated and weighed. Optimal timing is important as it can become difficult to randomize once an intervention is introduced in the community. Ethical considerations including informed consent, medical referrals, and appropriateness of experimentation should be evaluated. It is also important to assess the appropriateness of specific cohorts for a particular clinical trial question as the inclusion and exclusion criteria, outcome assessments, and other logistic factors could differ. Investigators involved in existing cohorts and registries should thoroughly evaluate, on an ongoing basis, the potential types of interventions that could be incorporated. Registries should also be examined for opportunities to conduct trials as they are typically larger than population-based cohort studies, more inclusive of “real world” patients, less expensive to assemble, and may be better for detecting safety signals. This report expands on an NIH workshop conducted to review these issues [1]. Specific examples from ongoing trials and observational studies are provided with recommendations for moving the field forward.

2. Experience with combined cohort studies and clinical trials

Several studies have been conducted that have leveraged existing or concurrent cohorts in their design. The experience with these studies is provided with specific examples. Each illustrates a unique approach.

2.1. The strong heart and the stop atherosclerosis in native diabetics studies

The Stop Atherosclerosis in Native Diabetics (SANDS), [2] is an interventional trial embedded in the Strong Heart Study, The Strong Heart Study (SHS) is a prospective observational study of cardiovascular disease in 4549 tribal members ages 45–74 years in North and South Dakota, southwestern Oklahoma, and Arizona. In SHS, rates of coronary heart disease (CHD) were higher than rates in other US populations; most of the events occurred in individuals who had diabetes, that is, half of the population [3]. The SANDS Trial tested whether the reduction of LDL-C, non-HDL-C and tighter control of blood pressure would have a positive effect on CVD risk in American Indians with diabetes. Five hundred American Indians from SHS with diabetes but no prior CVD diagnoses were randomized into 2 treatment groups. The first group was treated to the currently recommended levels of lipids and blood pressure (LDL-C < 100 mg/dl, non-HDL-C < 130 mg/dl and systolic BP < 130 mm/HG). The second group was treated to lower risk factor levels than were recommended at the time of the study (LDL-C < 70 mg/dl, non-HDL-C < 100 mg/dl and systolic BP < 115 mm/HG). After 3 years of intervention, both groups demonstrated a reduction in atherosclerosis progression as measured by changes in carotid intima-medial thickness (CIMT), with the more aggressively treated group showing regression of CIMT [4]. The approach of embedding the trial in the cohort study was cost-effective, as it used facilities and personnel already designated for the SHS study allowing completion in a more timely and economical fashion. In addition, it provided an important psychological boost to the large population of American Indians with diabetes who were witnessing a significant increase in CVD morbidity and mortality prior to the development of more aggressive risk factor management regimens.

2.2. The cardiovascular health study and the Ginkgo Evaluation of Memory Study

The Ginkgo Evaluation of Memory (GEM) Study was a double-blind, placebo controlled trial of Ginkgo biloba to reduce incidence of dementia. Secondary endpoints were cardiovascular diseases. There were 3072 participants aged 75+ in the four centers in the United States [5, 6]. The initial plan was to recruit participants from the Cardiovascular Health Study (CHS). Prior to the start of the trial, CHS participants were questioned as to willingness to participate and a high number were interested in the trial. However, of the 2409 potential GEM participants from CHS, only 249 (10%) were recruited to the GEM Study. The 4 centers that participated in the CHS were very successful in using targeted mailing lists (243,400) to successfully recruit the additional GEM Study participants within the allotted time [7]. There were three important lessons: 1) it is difficult to recruit from a long-term, ongoing, longitudinal study after participants are no longer being evaluated in the clinic; 2) participant response to questionnaire about willingness to participate in a trial and actual participation may be discordant; and 3) large data bases available in experienced clinics provided a successful backup for recruitment, even for older participants.

2.3. The Jackson Heart Study and the Health Promotion Study

African-Americans are recognized to carry an excess burden of cardiovascular disease (CVD) [8], and early results of the Jackson Heart Study (JHS) confirm high prevalence of traditional and putative risk factors for disease [9–11]. These facts stimulated intense interest among the JHS participants and JHS investigators in interventions that can impact outcome and alter the widening gap of mortality and morbidity between African-Americans and other groups. However, methodological concerns related to possible confounding of study observational outcomes have inhibited the “nesting” of clinical trials within the JHS cohort in the past. The Health Promotion Study is a pilot study testing the feasibility of a yoga intervention vs. regular walking and counseling among a cohort of middle-aged to elderly African Americans participating in the JHS, with a planned full-scale study to follow. The non-pharmacologic, non-invasive (and potentially homeopathic) nature of the intervention facilitated its approval by the JHS Steering Committee and the National Heart, Lung and Blood Institute (NHLBI), thus breaking the non-interventionalist history of the JHS. These same features are at least partly responsible for the rapid recruitment for the study. But perhaps the most important factors influencing participation in this trial were 1) the high level of trust of JHS and JHS-related activities among the cohort, along with 2) a high level of interest among cohort members in any novel efforts aimed at lowering their personal and familial risk of disease. Thus, of the 492 JHS participants prescreened (approximately 10% of the active JHS cohort) for eligibility, 438 were found to be eligible; only 6 participants refused to be prescreened. Of these, 382 participants have completed their baseline visit of whom 375 (86% of the eligible after prescreening) have been randomized. Study is underway to determine retention and adherence rates in each intervention arm: yoga (1, 2 or 3 times per week); a walking-based exercise program; and a health education-only program. The results will provide insight into the feasibility of non-traditional approaches as possible adjuncts to usual care for persons at risk for CVD. The JHS study example was more successful than the CHS and we believe that the following were key factors: 1) the intervention was of interest to the study participants, 2) the cohort was still engaged in active follow-up and 3) the number of adverse health events had not yet begun to accumulate.

2.4. Embedding intensive behavioral therapy in a clinical setting

Intensive behavioral therapy (IBT) for obesity can be considered an intervention to embed in observational studies. Implementation studies can also be conducted in real life, clinical settings which serve as the cohorts from which the participants for intervention are drawn, thus embedding a trial within a practice. The Social and Health Research Center implements the BuenaVida intensive behavioral therapy (IBT) for obesity in South Alamo Medical Group primary care clinics in San Antonio, Texas using a pre–post evaluation design [12]. The Centers for Medicare and Medicaid Services (CMS) began reimbursing outpatient clinic for providing IBT for obesity in March, 2012. The Healthcare Common Procedure Coding System (HCPCS) code G0447, along with one of the ICD-10 for body mass index (BMI) 30.0 and over (V85.30-V85.39), are used to bill for the service. The eligibility to participate is a BMI ≥30 kg/m². The BuenaVida follows the programming schedule set by CMS: one 15-min session every week during the first month; two sessions a month for the next two to six months; and one session a month for the next six months (18 sessions a year). A major problem with behavioral interventions is cost and sustainability. Because IBT for obesity is now reimbursed by several health insurance plans ($24.21 a visit), sustainability might be facilitated. The evaluation of the intervention may better reflect effectiveness in a real world setting.

2.5. The Women’s Health Initiative Observational Study and Clinical Trial

The Women’s Health Initiative (WHI) is an example of a hybrid design from initiation [13]. The WHI involved 68,133 women recruited into one or more of the four clinical trials, consisting of two trials of post-menopausal hormone therapy, a trial of low-fat dietary modification, and a trial of calcium and vitamin D supplementation, as well as 93,676 women who enrolled in the observational study. Participants were recruited from the communities surrounding the 40 WHI clinical centers. Eligibility for all components included age 50 to 79 years old, postmenopausal, and expected survival and local residency for at least 3 years. Women who were excluded from specific trials for reasons of safety, adherence or competing risk were offered enrollment in the observational study. Major outcomes in the trials included coronary heart disease, breast cancer, and hip fracture. The observational study goals were to explore the predictors and natural history of important health problems in postmenopausal women and to serve as a secular control for the clinical trials. Initial recruitment efforts were devoted to the dietary modification and hormone trials. For the observational study, there were two paths to enrollment with about half being accrued from each source: interested but ineligible for one of the trials or unwilling to be randomized and direct enrollment into the observational study. Women enrolled between 1993 and 1998, and continue to be followed.

3. Pros and cons of hybrid observational and clinical trial design

There is a need for more rapid translation from observational studies to clinical trials. Intervention studies may be initiated in a more timely fashion if embedded in an existing study. This could foster more rapid testing and translation of new prevention and/or treatment strategies. The state of the science for individual study questions should set the timing for the introduction of interventions in observational studies vs. new studies. As observational studies demonstrate the importance of a risk factor for an outcome, more timely translation to intervention studies is needed. The significance of the research question and the potential impact of the question on the health/disease status of the observational study population should determine whether the intervention study would be best conducted within that observational (cohort) study. In cases where the cohort has been followed for a long time and health outcomes have already occurred, a new hybrid design (clinical trial with cohort component) with recruitment of a new study population may be needed.

Timing is critical. Testing interventions related to a new risk factor or involving a new technology should be done early enough before the treatment or new technology of interest are widely adopted. Once new treatments or technologies are widely utilized in the community, especially when paid for by third party payers, insurance companies, etc., the ability to do trials is greatly limited because of the substantial “crossovers” within the trial, i.e., individuals obtaining diagnostic techniques or therapies from other sources than the trial. Timing considerations also are relevant to observational studies; once studies are established, participants may be less willing to be part of a new study that requires more visits or procedures. The experience with recruiting from the CHS study for the GEM trial is one example of this. There might be trade-offs between the needs of the intervention study and the observational study. For example, it might not be possible for the observational study to have enough power for its aims if the intervention substantially alters the natural history of the condition. In this situation, the intervention study needs to be important enough for embedding to be allowed to occur. Should a clinical trial be added, the investigators should monitor its impact on the outcomes and adverse events of the observational study.

Embedding an intervention study in a cohort study can enhance external generalizability and calibration of risk estimates. Randomized clinical trials (RCTs) are designed to address very specific hypotheses; a single intervention tested in a well-defined study population to provide definitive results for one primary outcome or at most a small number of designated outcomes. Even when the trial results are crystal clear, a large number of questions often arise: Do these results apply to other related interventions (similar drugs, related screening strategies, alternative behavioral interventions) or can we determine the critical component of an intervention that is driving the findings (specific aspects of diet or of exercise patterns, weight loss or cardiopulmonary fitness)? Do trial results generalize to populations that were excluded from the trial, perhaps for access, feasibility, cost or other factors that are difficult to manage within the trial itself (e.g., those with co-morbidities)? Are there other important outcomes, beyond those targeted in the trial, that are affected by the intervention for which the trial itself was not powered to detect? When trial results contradict prior observational studies or practice, questions of generalizability become even more prominent.

Our research enterprise does not have the wherewithal to mount additional trials to answer even the most important questions raised in this context. Further, in some instances the ethics of doing so would be questionable. But these are important questions that can be anticipated and in some instances addressed in a cost-efficient manner by embedding a RCT in a broader observational study or registry. If the intervention or related interventions are already in use in the general population, a resource that amasses the data in this observational setting in a parallel fashion could be used to extend the results of the clinical trial.

Such an approach has been used to considerable advantage in the WHI to elucidate the effects of hormone therapy on chronic disease risk. In a series of articles, Prentice and colleagues, [14–19] jointly analyzed data from the two WHI hormone trials and the parallel WHI observational study, demonstrating that only a small fraction of the discrepancies between prior observational studies and the RCTs was explained by traditional confounding. The more important source of these differences arose from the time-dependent effects of hormone therapy, often missed in observational studies, and the timing of initiation of hormone use relative to menopause (gap time). Similar joint analyses improved the power to examine subgroup analyses of the hormone trials and the WHI Calcium and Vitamin D trials [20] and to analyze other endpoints for which the trials were not adequately powered. Fundamental to these analyses was the comparability of the underlying study population (recruited simultaneously from the same communities) and the data collection in both study components, limiting methodological disparities [21].

It is important to note that there are great efficiencies that can be realized from hybrid designs. Potential advantages accrue from pre-existing, well-characterized, and engaged populations that are already under active follow-up. Very simple interventions such as randomizing the reporting of screening and follow-up should be considered. Pre-existing procedures for case finding, retention, outcome ascertainment and event classification enable rapid trial initiation and obviate the need to separately establish (and fund) infrastructure for trial-related procedures.

Randomized drug/comparative effectiveness research/safety trials should be of sufficient impact and quality to contribute to evidence-base for public health and clinical practice guidelines. The availability of data with limitations in quantity or quality should not preclude proposing and designing well powered, definitive studies. Researchers should recognize that some cohort studies may not offer sufficient sample size, particularly if effect sizes are small and/or there are multiple inclusion/exclusion criteria. Additionally, it is important to recognize that cohort studies also have exclusions and selection bias that must be considered in determining generalizability of a hybrid design.

To determine the benefits and risks of adding a clinical trial component to an observational study, clinical trials experts need to be included on the team. Clinical trials experts can work with observational studies researchers to optimize trial design, infrastructure needs, statistical issues, screening and recruitment, informed consent, adverse-event reporting, event definitions and adjudication, and strategies on retention and adherence. Conversely, observational studies researchers can help clinical trialists achieve efficiencies by drawing on their own staff, infrastructure, follow-up, case finding, and event classifications procedures.

4. Examples of hybrid designs based on registries and comparative effectiveness

Randomized clinical trials suffer from uncertainty about generalizability to broader populations, expense and limited power, especially in subgroups. Randomized trials may also not address questions of societal interest. Thus, non-randomized approaches using data from observational databases can be used to address questions of clinical interest. However, the problem with comparing therapeutic or diagnostic strategies with observational data is residual treatment selection bias due to unmeasured confounders, measurement error or bias in surveillance. An area of continuing interest is the choice of revascularization strategy for stable ischemic heart disease. In particular, questions remain concerning the comparative effectiveness of percutaneous coronary intervention (PCI) and coronary-artery bypass grafting (CABG).

The American College of Cardiology Foundation (ACCF) and the Society of Thoracic Surgeons (STS) have developed a partnership, the ACCF and STS Database Collaboration on the Comparative Effectiveness of Revascularization Strategies (ASCERT), to compare the outcomes of PCI and CABG, using information from records in their respective societal databases, with follow-up data from claims records of the CMS. In ASCERT, the ACCF National Cardiovascular Data Registry and the STS Adult Cardiac Surgery Database were linked to claims data from the CMS for the years 2004 through 2008. Outcomes were compared with the use of propensity scores and inverse-probability-weighting adjustment to reduce treatment selection bias. Among patients 65 years of age or older who had two-vessel or three-vessel coronary artery disease without acute myocardial infarction, there was no significant difference in adjusted mortality between the groups (6.24% in the CABG group as compared with 6.55% in the PCI group; risk ratio, 0.95; 95% confidence interval [CI], 0.90 to 1.00) at one year. At 4 years, there was lower mortality with CABG than with PCI (16.4% vs. 20.8%; risk ratio, 0.79; 95% CI, 0.76 to 0.82) [22]. The possible influence of residual confounding was assessed by means of a sensitivity analysis. In this observational study, we found that among older patients with multivessel coronary disease that did not require emergency treatment, there was a long-term survival advantage among patients who underwent CABG as compared with patients who underwent PCI. These data are largely consistent with clinical trials though they do not address medical therapy which is also effective [23]. Thus, comparative effectiveness studies using data from large registries can be used to address issues of societal interest, but the problem of treatment selection bias remains.

Large national registries have been developed to evaluate specific disease outcomes but also the processes and infrastructure of patient care. Observational data from registries can extend the value of clinical trials because they are more inclusive of real-world patients and can include very large cohorts at less expense. Observational data from registries can be hypothesis generating and help detect safety signals for rare events. For example, fatal stent thrombosis is rare but detectable through registry follow-up [24]. Registry infrastructure offers multiple operational efficiencies to both study coordination and to sites participating in RCTs. The study of access sites for enhancing PCI in women showed a 65% reduction in workload for site coordinators using an ongoing registry for recruitment [25]. Alignment of objectives and operational structure with federal public health, regulatory and reimbursement science promotes a collaborative and inclusive and cost effective approach for all stakeholders interested in uncovering new device indications and best practice guidelines.

5. Pros and cons of registries

Registries offer researchers large populations of well-characterized patients with a wide variety of diseases, conditions, and interactions with clinical care. Researchers should consider linking clinical registry databases with administrative databases, in particular for enabling efficient recruitment, screening, and follow-up. Limitations of registries may include lack of a biorepository, disease-specificity that may render them less valuable for primary prevention trials, potential need for additional data collection of data outside the administrative data or medical record (such as quality of life, adverse events, or adherence), the lower quality and completeness of administrative data and challenging requirements for statistical methodologies.

Observational registry studies were not designed to test the efficacy or even effectiveness of specific therapies or technologies, but have their primary value in monitoring quality and quantity of outcomes and signals for unexpected adverse or even potentially beneficial effects, especially within specific disease subgroups defined on genetics, demographics, or health-related characteristics in non-randomized comparative effectiveness studies. In this way, registries can assess the consistency of the findings of clinical trials in populations not represented in a RCT. While registry studies can be large, size does not overcome treatment selection bias. Specific methods must be used to limit such biases. These include restriction to indication, application of inclusion and exclusion criteria at “time zero” and using an intention-to-treat approach [26].

6. Need for partnerships

Observational studies have often gained broad community engagement to support the need for long term follow-up. If an intervention is considered for an observational study, researchers should strongly consider engaging pertinent stakeholders, including the study participants and their communities, in the formulation of the research questions, the oversight, and implementation of the intervention study. Diversity and inclusiveness are important principles. Many of the populations in need of new approaches to care may be more difficult to reach, but should be included in efforts to conduct both observational and clinical trial research.

Both epidemiological and clinical studies that intend to generate findings and interventions that would generalizable to a large proportion of the population require adequate representation of those most at-risk of disease and susceptible to disparities. The recruitment of minority and underserved individuals into research studies is often perceived as challenging or problematic by many, [27] whereas others have found that individuals from these groups are more open to participating in research that often thought [28].

The spectrum of challenges that may be encountered during recruitment and follow-up of participants from underserved communities ranges from staff issues (for example, number and qualifications), [20] administrative issues (for example, identifying appropriate space where studies could be performed, complying with all regulations), [20] target population issues (inclusion and exclusion criteria; research question not resonant with the target population’s culture and perceived needs; high no show rate to appointments; individual’s distrust of the research and medical system) [29] and investigator’s own issues (for example, beliefs and perceptions about the target population and the scientific relevance of the question of interest) [30].

Strategies to overcome potential challenges may include hiring of staff who are familiar with or proficient on the target population’s culture and able to communicate in the same language; [31,32] recruiting participants from clinical settings; [32] inviting physicians to refer participants; [27,33] reaching out to community partners and involving community key stakeholders in the identification of the relevant research questions, design of the recruitment and follow-up plan; [34,35] communicate clearly and with veracity; [31] and one of the most fundamental strategies is to formulate a research question and study design that are relevant to the target population [33].

7. Summary of recommendations

Based on the considerations discussed above, the authors agree that established observational registries and cohort studies offer an infrastructure essential for embedding clinical trials, and specific opportunities for the implementation of this hybrid model should be developed. Building on the existing infrastructure would foster more rapid translation of findings from observational studies or registries into clinical applications. At the same time, the significance, impact and timing of the research question will determine whether the intervention should be implemented in an established observational study. Ethical aspects, such as obtaining consent to participate in the trial, appropriateness of the intervention in the study population, adverse events and medical referrals need to be considered. Finally, resonance of the research question with the population or community of interest and partnership with stakeholders (i.e. researchers, health care providers, community leaders, funding agencies, health policy and insurance experts) are fundamental for the success of the implementation of this model.

Abbreviations

SANDS

Stop Atherosclerosis in Native Diabetics Study in the Strong Heart Study

GEM

The Ginkgo Evaluation of Memory Study

CHS

Cardiovascular Health Study

CVD

cardiovascular disease

JHS

Jackson Heart Study

IBT

intensive behavioral therapy

CMS

Centers for Medicare and Medicaid Services

HCPCS

Healthcare Common Procedure Coding System

ICD-10

10th revision of the International Statistical Classification of Diseases and Related Health Problems

BMI

body mass index

WHI

Women’s Health Initiative

RCT

randomized clinical trial

PCI

percutaneous coronary intervention

CABG

coronary-artery bypass grafting

ACCF

American College of Cardiology Foundation

STS

Society of Thoracic Surgeons

ASCERT

ACCF and STS Database Collaboration on the Comparative Effectiveness of Revascularization Strategies

CI

confidence interval