Comparative effectiveness research in pediatric respiratory disease: promise and pitfalls

Abstract

Comparative effectiveness research (CER) has become increasing central to clinical research in medicine. CER seeks to conduct clinical trials that compare different commonly used interventions in real world settings (pragmatic clinical trials) and use a multitude of sources of evidence (including registries and cohort studies) to inform clinical decision making. CER also ensures that stake holders (patients, families, care providers, insurers) have a voice in the research process by integrating a formal stakeholder engagement as part of the research. This innovative approach to clinical research has distinct benefits and pitfalls. This review will first define what comparative effectiveness research is and then describe some of its benefits and then pitfalls. The focus will be on the role of CER in pediatrics.

1. Introduction

Clinical researchers are increasingly considering comparative approaches to clinical management and studies of therapeutic effectiveness outside of the tightly controlled clinical trial settings. These initiatives have been labeled CER. In many aspects, researchers have conducted CER without having a clear name for it [1]. CER now refers to specific types of clinical research that prioritize the ability to replicate or maintain real life clinical practice as a key element of the research design, which is fundamentally different from most traditional clinical research studies [2]. The Institute of Medicine (IOM) has deemed pediatric respiratory disease a key priority area for CER [3]. In this commentary, we will first define CER and then provide examples of its implementation in pediatric research. We will then describe many of the pros and cons of CER while highlighting the value of this approach to current and future pediatric research studies.

2. Definition

A number of proposed definitions for CER have been put forth (Table 1) [4, 5]. These definitions share common components: 1) research in ‘real world’ settings with comparisons of therapeutics or treatment approaches that are relevant to clinical decision making; 2) valuing diverse sources of evidence, including observational data; 3) prioritizing approaches that lead to broad applicability and dissemination of results. CER extends our understanding of evidence based medicine by shifting greater focus toward key stakeholders – patients, providers and payers. Continuous engagement with stakeholders is used to refine the research priorities and ensure that the research being conducted will serve their needs. Engagement of stakeholders encompasses identification of research topics, establishing priorities, methodologies and data syntheses [6-9]. In pediatric research, stakeholders include both patients and caregivers, as well as providers, pharmaceutical industry, insurers, government agencies and often supportive foundations [10]. One common approach in CER is to formulate research questions using the PICO framework: a Patient/Problem, Intervention, Comparators, and Outcome [11].

Table 1.

Definitions of CER noting differences.

	Institute of Medicine (IOM)	Department of Health and Human Services (DHHS)	Agency for Healthcare Research and Quality (AHRQ)
Research Defined	Direct comparison of effective interventions	CER is the “conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in ’real world’ settings.	CER is designed to inform health-care decisions by providing evidence on the effectiveness, benefits, and harms of different treatment options.
	Study of patients in clinical care with aim to tailor decisions to the needs of individual patients through the study of more heterogeneous groups of patients.
	Primary interests are disease prevention, systems of care, drug therapies, devices, surgery, and monitoring of disease.	Relative strengths and weaknesses of various medical interventions.	The evidence is generated from research studies that compare drugs, medical devices, tests, surgeries, or modes of health care delivery
		Research used to make decisions to improve the performance of the U.S. health care system.
	Synthesis of existing data and aspects of care delivery.
Study Description	Researchers must select amongst these methodologies: (1) Systematic reviews guiding development of guidelines; use of established data sets (2) Prospective registries and cohorts (3) Randomized controlled trials (4) Pragmatic randomized trials	Requires the development, expansion, and use of a variety of data sources and methods to assess comparative effectiveness.	(1) Existing data (systematic reviews of existing evidence). (2) Create new data: researchers conduct studies that generate new evidence of effectiveness or comparative effectiveness of a test, treatment, procedure, or health-care service.
			Goal is to support health services research that will improve the quality of health care and promote evidence-based decision making.
Stakeholder Description	Perspective is one of overall societal good. However, consumers, patients, and caregivers as well as their health care providers must be involved in all aspects of CER.	Patients, clinicians, and other decision-makers. Generation of knowledge for the research community is specifically noted.	Clinicians, consumers (payer, purchaser), and policymakers.
			Have a community forum to address different stake holders.
			Also uses the Healthcare Horizon Scanning System to identify new and emerging issues for CER investment.
Role in Disseminati on of Information	The nation will need effective strategies for disseminating CER findings and promoting their adoption by clinical practice.	Actively disseminate the results. Create the CER inventory.	Dissemination of the results in a form that is quickly usable by clinicians, patients, policymakers, and health plans and other payers
	Must inform the public about their methodological advantages and shortcomings
Addresses Cost as Outcome	Mentioned but is not a focus of recommendations	Not addressed	Not addressed.
Resources and Workforce	Requires increasing the need for trained experts in biostatistics, epidemiology, systematic reviews, observational and clinical trials, and more refined research methods for CER.	Development, expansion, and use of a variety of data sources and methods to assess comparative effectiveness.	Requires the development, expansion, and use of a variety of data sources and methods to conduct timely and relevant research
Specified Outcomes	Not specified	Comprehensive array of health-related outcomes for diverse patient populations and subgroups	Not specified

CER supports using a variety of evidence including administrative data analysis, systematic reviews, cohort studies, and case control studies (Table 1) [4]. While these clinical research study designs are not new, CER affords them greater weight than in the traditional hierarchy of evidence based medicine [12].

One of the more critical components of study design in CER is the emphasis on pragmatic or practical clinical trials compared to conventional efficacy trials. In pragmatic clinical trials, we assess effectiveness rather than efficacy [2, 13, 14]. Most clinical trials are efficacy trials – assessing whether the therapy or treatment approach works in a tightly controlled setting. Pragmatic clinical trials of effectiveness assess whether the therapy or treatment approach works in the broader population with few interventions other than usual clinical care to promote adherence to a protocol [15, 16]. CER research studies strive to provide information that can be generalized to a broad population affected by the disease or condition of interest.

CER is also characterized by a relatively low intensity of follow-up. In the extreme, investigators conducting a comparative effectiveness study do not interact with the participants outside the confines of usual practice, obtaining data on study outcomes from the medical record alone [13]. CER studies also emphasize active comparator studies without the use of placebo arms. In addition, the rigorous follow-up that is characteristic of an efficacy trial can both limit real-world applicability and potentially influence the observed effect size. Traditional efficacy studies typically limit study enrollment to patients with the disease of interest and without several excluded comorbidities in order to optimize statistical power and the assessment of benefit vs. risk of participation. In contrast, because an imperative of CER is to be readily generalizable, CER studies have minimal inclusion and exclusion criteria allowing enrollment of a more broadly representative cohort. Often the inclusion criteria for CER studies mirror common clinical criteria to identify target patients for studies. As an example, in a CER of pediatric asthma, physician-diagnosis of asthma may be used for inclusion rather than measures of lung function and bronchodilator effect.

One of the better examples of the differences between efficacy and effectiveness clinical trials can be observed in the study of patients with mild persistent asthma. Efficacy trials strongly support the use of inhaled corticosteroids as first-line agents in patients with mild persistent asthma [17]. Interestingly, when a pragmatic effectiveness trial was conducted comparing leukotriene-receptor antagonists to inhaled steroids, the authors noted that leukotriene-receptor antagonists were as effective as inhaled corticosteroids for long-term symptom control [18]. These findings were likely due in part to poor adherence with inhaled steroids for the treatment of mild persistent asthma. On a population level, a once-a-day pill like the leukotriene-receptor antagonists may be as or even more effective than a twice per day inhaler with spacer intervention due to improved adherence with simple drug regimens [19].

3. Benefits of CER

One of the true innovations in CER is the inclusion of stakeholder groups throughout the research process. The stakeholder groups have been variably defined but are generally quite broad, including: consumers, clinicians, researchers, policymakers, industry representatives, private and public health care purchasers, and health care leaders [10]. Such engagement can enhance participation in research. By integrating stakeholders into the research priorities, one helps ensure that prioritization emphasizes 'real-world’ research questions that are those most important to the communities impacted by the research.

Another great benefit of CER is the increased likelihood that results of pragmatic trials will reflect and inform real life clinical care decisions. Prior research in patients with cystic fibrosis (CF) has demonstrated that those patients who participate in classic efficacy clinical trials do not closely reflect the broader patient population [20]. The patients who participated in clinical trials had worsened clinical status, but less lung disease progression over time [20]. Given these differences, results from clinical efficacy trials might not generalize to the greater CF population, which have dissimilar comorbidities, access to care, or adherence to recommended therapies. One would like to demonstrate effectiveness in the entire target population to justify both expense and, in some cases, risk of therapies or treatment approaches. One recent example of a pragmatic CER study was the Early Pseudomonas Infection Control (EPIC) trial in children with CF [21, 22]. In this trial, researchers responded to an express desire from the CF community (practitioners, family members and funding agencies) to design a randomized clinical trial to assess different approaches to eradicate Pseudomonas aeruginosa from the airways of CF children who had recently acquired the organism. A placebo arm of the trial was rejected by the stakeholder community while four comparative treatment options were included in the study design. Inhaled antibiotics were given either on a set schedule or only when respiratory cultures were positive for Pseudomonas aeruginosa [22]. Both interventions were deemed acceptable by the CF community. The inclusion criteria were quite broad and only excluded subjects in which the treatment medications were deemed to be potentially harmful. The study identified that fewer antibiotics could be used to achieve the same result of P. aeruginosa eradication [22].

Another great advance in CER is the increased flexibility to evaluate a broader range of evidence to establish effectiveness. Evidence can come in the form of secondary data analyses of administrative data sets or of patient registries. Populations studied with both administrative data and disease registries represent a broad spectrum of patients who are managed primarily in real world conditions. These analyses allow one to study treatment effects that require longer term observation than is typically feasible in clinical trials (like survival or annualized change in health outcomes over several years). Organizations for rare diseases, like CF, have developed large scale patient registries that include data related to demographics, clinical status, and medication use, and provide a key resource for investigating CER questions. The Cystic Fibrosis Foundation has maintained a longstanding patient registry [the CF Foundation Patient Registry (CFFPR)] to track clinical outcomes and clinical care; the data for this registry have gone through formal validation for key variables [23] and are entered soon after clinical encounters to document how care is provided in the current era. This registry has been used to assess the long-term treatment effects of therapies in a broadly representative population of CF patients and has been able to demonstrate better survival associated with use of specific therapies [24]. Such outcomes would not be feasible in traditional clinical trials. Registries have also been instrumental in evaluating outcomes in CF between countries. One recent study of patient registry data demonstrated that US CF children have better lung function than UK children [25]. Differences in care and the early adoption of therapeutics for children in the US may account for these differences. In a more recent analysis, CF patients living in Canada were demonstrated to have a 10-year survival advantage compared to the US. This may be partly attributable to differences in how the therapeutic option of lung transplantation is utilized in the US compared to Canada [26].

These examples merely use data captured in each country’s respective disease registries. However, there are alternative approaches that exploit administrative data sources maintained by providers and payers of health care in order to identify patients with target conditions. The national effort to develop electronic medical records provides further opportunities to develop methodologies to identify patients [27]. The content of these data systems is expanding beyond diagnostic coding to include medications, laboratory values, cost, provider level characteristics, and free text. Such data sources can now be used to deploy machine learning to assess the impact of different therapeutic and treatment approaches [28].

4. Pitfalls in CER

Rarely do decisions in clinical research come without potential pitfalls. CER has its own set of key pitfalls. One of the most challenging areas is that of the pragmatic clinical trial. If efficacy has not yet been established for a specific intervention through traditional clinical trials, using a pragmatic trial design risks weakening one’s ability to demonstrate that the therapy works as intended. In this scenario, a failed pragmatic trial may establish that the therapy was ineffective in a heterogeneous population but would not determine if it has efficacy in a target population. This approach could prematurely or erroneously halt the development of a drug or therapeutic approach. A great example of this phenomenon occurred in asthma therapy; many of the novel biologics for asthma (i.e. mepolizumab failed when evaluated in larger heterogenous populations of asthma patients [29]) that have now been approved by the Food and Drug Administration for specific sub-phenotypes of asthma. Thus, developing a drug using a personalized approach to therapeutics is in conflict with CER.

Another potential pitfall of CER is that research priorities must emanate from stakeholders. While this approach can prioritize research that the community wants, it may limit the ability of researchers to tackle controversial, challenging, or highly novel clinical research topics. Intense stakeholder involvement may add additional barriers to conducting ground breaking research that is not understood to address one of the community’s greatest needs. Many scientific discoveries have emanated from underappreciated theories or study proposals that the community would reject. A great example of such a research development was that of the discovery of H. pylori as the causative agent of the majority of peptic ulcer disease [30]. The greater community of academics rejected the H. pylori hypothesis, and one could easily envision stakeholder perspectives that could limit scientific investigation [31] in other fields.

CER is based on the premise that comparing efficacious therapies (active comparator trials) is an integral component of the research. This approach can fail to consider changes in the target patient population over time that may develop with increasing or long term exposure to treatment. Many active comparator studies require a non-inferiority design rather than a superiority design, which is susceptible to a concern termed “bio-creep” [32, 33]. A scenario may develop when additional therapeutics or therapeutic approaches are slightly less effective than the prior treatments but deemed non-inferior. As additional treatment options are considered, they may become less and less effective but remain non-inferior in active comparator trials. If one were to compare such options to placebo, it could be no better than placebo due to bio-creep. Thus, a research strategy that emphasizes active comparator studies using non-inferiority designs (as seen in CER) can be problematic. Superiority trials can overcome this important limitation of serial non-inferiority studies.

Clinical trials typically rely heavily on clinical care outcomes (e.g. survival, health events, physiologic markers, etc.) that work well for clinicians, but often fail to specifically address issues that patients and their families may find central to their needs, like symptoms and treatment burden [34]. CER focuses on patient centered outcomes. One concern that could be raised about CER is its strong emphasis on patient centered outcomes. A study may result in improved patient centered outcomes without improvement in or even worsening of key classically derived clinical endpoints. One could take the extreme position and state that the use of cocaine, for example, improves patient centered symptoms and patient reported quality of life; such a therapy could theoretically be deemed an effective therapy to achieve these therapeutic aims but is this drug a good treatment?

The last major challenge with CER as currently defined is the complete absence of economic analyses integrated into the assessment of therapeutic benefit. Cost effectiveness analysis (CEA) is an integral component in the development of clinical guidelines in other countries including Australia, England, and Canada [35]. This exclusion of cost in CER does limit the use of this data in setting health policy and thus limit the implications of the research. However, the prospect of using this information to make decisions is controversial [1, 36, 37]. A compromise is emerging where measuring resource use is desired as long as the combination of costs and health outcomes (e.g. CEA) is not explicitly used to make decisions [37]. Evolution of healthcare in the US market place may have longer term influences on cost evaluations if healthcare costs continue to escalate.

With CER as a growing research modality for population-based assessments, one must consider the potential conflict with personalized medicine wherein the goal is to provide tailored treatments to individuals [38, 39]. CER provides data on ‘average population effectiveness’ of a particular treatment relative to another, but as variation can exist for any treatment at an individual level, potential benefits of personalizing a treatment are ignored. Although secondary and subgroup analyses can mitigate some of this potential variation, the ability to adjust for individual-level confounders is limited by the availability and granularity of the datasets used [40]. Conversely, given the current era with limited resources and budgets, CER data can be used to inform personalized medicine such that any investments will yield therapies with significantly improved health outcomes, and in this way act in a complementary manner [40].

5. Conclusions

CER appears to be here to stay. This avenue of clinical research has certain real strengths over the more traditional approaches to clinical epidemiology and clinical research. The structure of CER pushes researchers to engage the community and stakeholders early in the development of a research proposal. The transition from efficacy to effectiveness research can give the clinical community a clear vision of how therapies and treatment approaches work in the real world. This data will help support clinicians in their goals to appropriately inform their patients about the likely treatment outcomes that go beyond the results seen in tightly controlled double blind randomized controlled trials that are the center piece of efficacy based clinical research. Both CER effectiveness trials and classical efficacy trials can and should exist together. Each research approach addresses different research questions and needs to be included in the clinical trial compendium. Ultimately the choice of study will depend on the questions to be addressed.

Key Points.

Comparative effective research is a relatively new approach to clinical research that engages stakeholders, leverages multiple types of health information and supports the conduct of research in real world settings, all to improve the delivery of health care. There are key strengths and weaknesses of comparative effective research as it applies to pediatric research. Ultimately, providing a broad spectrum of data to help guide medical decision making and engaging those impacted by those medical decisions is a clear advance for healthcare.