A call for action: comparative effectiveness research in asthma

Abstract

Comparative effectiveness research (CER) has received considerable research attention in recent months, and efforts to promote CER are part of the newly enacted Patient Protection and Affordable Care Act. In this paper, we define CER, how it complements traditional efficacy research in asthma, and discuss how CER can help provide the basis for rational decision-making about the care of individual patients with asthma and how best to deliver this care in real-world settings. We present information about the challenges and opportunities to conduct CER, including enhanced patient registries for observational CER and effectiveness trials (also called pragmatic trials). We discuss the urgent need to define the appropriate methodologies for CER and to develop and prioritize a research agenda for CER studies in asthma with the help of a diverse group of stakeholders.

Randomized controlled trials are the gold standard by which the benefits and harms of treatments have been established, because this approach greatly minimizes the risk of confounding and avoids selection bias. Such studies have traditionally employed an efficacy paradigm in which various approaches are used to amplify the signal-to-noise ratio in order to address the question “Can this intervention work?” Such studies are critical to the development of innovative approaches to patient care. A hallmark of the efficacy trials is the application of narrowly defined inclusion and exclusion criteria to select study subjects. A recent study found, for example, that a median of 6% (range 0 to 43%) of patients treated for asthma met eligibility criteria for major trials cited in evidence-based treatment guidelines.¹ A high level of selectiveness is employed in efficacy trials to ensure that study subjects are enriched with patients most likely to benefit and least likely to be harmed by the study interventions.

Another feature of efficacy trials is the conduct of such studies in a research environment that promotes uniform application of the intervention to study subjects, often by expert clinicians and in resource intensive settings. The National Heart, Lung, and Blood Institute-sponsored AsthmaNet, for example, is a multi-center network led by Principal Investigators with established research programs in pediatric and adult asthma and includes research staff with extensive experience in the conduct of clinical trials.² Traditionally, research environments for asthma clinical trials have included standardized training sessions for study staff and extensive and detailed participant instructions to increase treatment fidelity and subjects’ adherence to study procedures. Site-visits by study monitors are often used to reinforce staff adherence to study procedures during the course of the study.

Efficacy studies also typically employ an intensive schedule of study visits (generally longer in duration and more frequent than occurs during typical patient-clinician encounters) to measure the health status of study subjects at multiple time-points. Comparisons in efficacy studies are often selected to provide the greatest contrast between treatment strategies, most commonly by using inactive controls (e.g., placebos or shams).^3,4 There is less of an emphasis on head-to-head comparisons (active controls) between newer interventions vs. existing treatment approaches, though recent studies have increasingly adopted use of active controls.⁵ Primary endpoints in efficacy studies have traditionally emphasized surrogate markers of disease severity or progression (e.g., lung function measured by peak expiratory flow rates or spirometry, or rate of change in lung function), rather than outcomes that are clinically meaningful to patients (patient-centered outcomes), such as symptom control, burden of asthma on their activities, or need for additional treatments that have troublesome adverse effects (e.g., systemic corticosteroids, which can affect mood and sleep quality). More recent trials, though, have increasingly employed patient-centered outcomes as primary endpoints.³ The goals of an efficacy research environment are to minimize confounding and bias, to maximize the opportunity to detect an effect (difference in outcomes between comparison groups), and to avert, detect, and expeditiously manage adverse events in study subjects. Generalizability is generally lower priority in efficacy studies.

Analyses within efficacy studies indicate that patients may respond differently to identical treatments.⁶ It is also not uncommon to find patients in real-world clinical practice settings who do not accrue the benefits reported in efficacy studies or patients who have difficulty tolerating or using treatments shown to have a favorable benefit to harm ratio in efficacy studies. Explanations for this heterogeneity in treatment effects include differences in study subjects in characteristics that affect responsiveness, differences between study subjects in efficacy studies and patients in clinical practice, as well as differences in how interventions (as well as co-therapies, such as use of inhaled corticosteroids and avoidance of environmental triggers) are used by patients and by clinicians. Poor patient and clinician adherence to asthma therapy is common, likely because the multiple supports to promote treatment fidelity in research environments are generally absent in practice settings.^7–12 Patients in clinical practice therefore receive care in heterogeneous healthcare settings with varying levels of resources, which may also affect the extent to which harms of interventions can be averted or managed appropriately.

Such factors may modify the extent to which benefits outweigh harms of treatment in individual patients. Thus, therapies with a favorable benefit to harm ratio in efficacy studies may offer no net benefit in some groups of patients in real-world clinical practice settings; in other words, therapies with demonstrated efficacy may not be effective in some patient subgroups. In some cases, efficacious treatments may even result in net harm in some patients when applied more broadly in routine practice, as was shown in the large post-marketing clinical trial of salmeterol in patients with asthma and in case reports related to other asthma therapies.^13,14 Moreover, use of inactive controls in randomized clinical trials (e.g., placebo or sham), may not provide the information needed by patients and clinicians when considering existing (including non-medication) treatment alternatives. Comparative studies with active controls are needed in such cases.

These issues have led many to conclude that more comparative effectiveness research (CER) is needed. Indeed, recent healthcare legislation (Patient Protection and Affordable Care Act, 2010) calls for the creation of a new entity, the Patient-Centered Outcomes Research Institute (PCORI), which will greatly increase the visibility of CER as a critical component of the nation’s portfolio of biomedical research.^15,16

What is CER? As defined by the Patient Protection and Affordable Care Act, CER (or ‘comparative clinical effectiveness research’) is “research evaluating and comparing health outcomes and the clinical effectiveness, risks, and benefits of 2 or more… health care interventions, protocols for treatment, care management, and delivery, procedures, medical devices, diagnostic tools, pharmaceuticals (including drugs and biologicals), integrative health practices, and any other strategies or items being used in the treatment, management, and diagnosis of, or prevention of illness or injury in, individuals.” Importantly, The Patient Protection and Affordable Care Act recommends research to evaluate patient-centered outcomes and sources of heterogeneous treatment effects as key components of CER, including differences in patient characteristics and in models of healthcare delivery. The purpose of such research is to “assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations, and the dissemination of research findings …”

Thus, CER includes a range of activities (original research, evidence synthesis, evidence dissemination) about the comparative harms and benefits of existing pharmacologic and non-pharmacologic interventions that will help healthcare decision-makers choose the most appropriate intervention for individual patients and how best to deliver this intervention. In short, whereas efficacy studies are key to identifying innovative approaches to care and are designed to answer the question “Can this intervention work?”, CER studies are key to addressing questions about “Which of the existing therapies work best in individual patients and which is the best way to deliver this care?” The National Institutes of Health and the Agency for Healthcare Research and Quality (AHRQ) have begun to offer a growing number of opportunities for CER; such opportunities are expected to increase even further through funds distributed by PCORI, which are expected to increase over the next several years to about $500 million per year or more.

Options for CER include observational studies and effectiveness trials. Observational effectiveness studies involves tracking interventions and outcomes in real-world settings, often facilitated by electronic sources of administrative or claims data that are routinely collected for billing purposes. These can be cohort studies, in which outcomes are compared in patients receiving different treatments, or case-control studies, in which previous treatments (inhaled corticosteroids, yes or no) are compared in patients with versus without outcomes of interest (e.g., asthma-related death). Pharmacy dispensations and healthcare utilization data (e.g., emergency department visits) are usually reliably captured in these studies. Administrative data have historically lacked some important clinical information (e.g. symptoms, laboratory data, pulmonary function, smoking status), some types of healthcare interventions (e.g., patient education, smoking cessation counseling), and the rationale used in healthcare decision-making (e.g., patient preferences, intolerance). The lack of such data introduce multiple sources of unmeasured confounding and selection bias in observational effectiveness studies based on administrative data alone.

With greater use of electronic medical records, the opportunity for increasingly rigorous observational CER studies based on linked administrative and clinical data sources (‘enhanced’ patient registries) has never been greater. Clinical information for enhanced registries can also include data prospectively collected from patients (e.g., asthma symptom frequency, health-related quality of life, self-management practices, depressive symptoms and other psychosocial characteristics). Enhanced registries thus provide the opportunity for designing studies and analytical approaches that reduce confounding and selection bias. For example, pulmonary function data in electronic medical records, together with prospectively collected information about symptom frequency and medication use, could be used to create more refined categories of asthma control than categories that rely exclusively on administrative data about patterns of healthcare utilization. When different healthcare settings and geographic regions are included in patient registries, observational CER studies provide unparalleled opportunities to describe ‘usual care’ and to identify clinically relevant comparators. Observational CER studies are uniquely suited for large-scale studies with the statistical power to address clinically meaningful outcomes in real-world settings, including rare but serious adverse events (e.g., asthma-related deaths).¹⁷

Rigorously performed observational CER thus provides the opportunity for answering questions about the effectiveness and safety of specific interventions (or specific groups of interventions) with a high level of generalizability. Well done observational CER studies are neither necessarily quick nor inexpensive; the resources needed for observational CER will depend on the specific research question (i.e., adequate characterization of the population(s), practice setting(s), intervention(s), comparator(s), and outcome(s) in whom the results will be applied) and whether prospective data collection is required. Enhanced patient registries that are used for observational CER studies also can be re-purposed to provide the data sources for measuring changes in practice and outcomes following the implementation of quality improvement initiatives.

However, observational CER studies suffer from the limitations of all observational research– the inability to fully account for selection bias and unmeasured confounders, which only randomization allows. There is thus substantial interest in effectiveness trials, designed to rigorously test the harms and benefits of interventions in the clinical settings in which patient care occurs. Effectiveness trials (also called ‘pragmatic trials’) are primarily designed to evaluate the effects of a healthcare intervention under conditions in which it will be applied, whereas efficacy trials (also called ‘explanatory trials’) are primarily designed to evaluate the effects of an intervention under ideal circumstances.¹⁸ We find the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) tool to be a particularly useful framework for understanding the differences between efficacy trials and effectiveness trials, which we have adapted for asthma studies and provided in this paper (Table). The PRECIS tool (and our adaptation for asthma studies) is still at the conceptual stage and needs empiric validation, but emphasizes that trials cannot be simply characterized as being an effectiveness or efficacy study. Instead, each of the various decisions regarding the design of the study (e.g., selection of a study population, selection of the intervention and comparators, selection of the primary outcome) can move the trial conceptually along a continuum from efficacy to effectiveness for that domain. Some domains in a trial may be more consistent with the effectiveness paradigm (e.g., recruitment of study populations from community-based practices, rather than asthma research centers). Others may more closely resemble an efficacy framework (e.g., study conducted by investigators at specialized research centers for asthma). Because there are multiple domains that contribute to the design of a trial, the trial itself will fall along a multi-dimensional continuum of efficacy to effectiveness. Ideally, CER studies employing clinical trial designs should conform to the effectiveness end of the continuum in as many domains as possible and results should be reported using the recommendations of CONSORT guidelines for pragmatic trials.¹⁹

Table 1.

Effectiveness to efficacy continuum for clinical trials

Domain	Efficacy (or explanatory) end of the continuum		Effectiveness (or pragmatic) end of the continuum
1. Selection of patients	Stepwise selection criteria are applied to enroll subjects at high likelihood of benefit, low likelihood harm, high likelihood of adherence to study procedures (e.g., multiple exclusion criteria, such as more than 5 pack-years of smoking, age greater than 65 years, and unable to complete daily diary during run-in period). Generalizability is not a priority.		Generalizability is a priority. All participants with condition of interest (e.g., diagnosis of asthma regardless of smoking status, age, or ability to complete daily diaries). Specify subgroups in which heterogeneous treatment effects may be clinically important and needs confirmation.
2. Selection of experimental intervention(s)	High level of fidelity to study procedure encouraged (e.g., participant / clinician only given choice of continuing current and monitoring for adverse effects or premature study termination).		High level of flexibility, tailoring of intervention permitted (e.g., reduction of dose or temporary drug holiday due to minor adverse effects).
3. Selection of clinicians and practice settings to conduct study	Experienced clinician and practice settings with high rates of success and low rates of complications (e.g., expert clinicians with established research programs in asthma).		Full range of clinicians and practice settings, regardless of research expertise (e.g., specialists and non-specialists in community-based clinics and academic centers).
4. Selection of comparator	May use a placebo or other inactive control (e.g., sham procedure) rather than the best existing intervention in current use.		Usual care or best existing intervention in current use by clinicians (e.g., alternative approaches recommended by asthma guidelines).
5. Follow-up schedule	Study visits (separate from routine clinical encounters), typically longer in duration and more frequent than would occur in routine practice.		No formal follow-up visits of study individuals. Administrative / clinical data sources (e.g., electronic medical records, mortality registries) are searched for outcomes.
6. Selection of primary outcome	Often clinically meaningful outcome to patients, but may be surrogate marker of an outcome of interest (e.g., change in lung function, airway hyperreactivity).		Clinically meaningful outcome to study subjects (measures of health / well-being easily recognized by patients, such as number of courses of systemic corticosteroids, missed school or work).
7. Adherence of patients and clinicians	Adherence monitored closely and rigorously (e.g., electronic medication monitors, electronic daily diaries), may be prerequisite for study entry; strategies to promote adherence employed.		Unobtrusive (e.g., patient self-report) or no measurement of participant adherence. No study support to reinforce or improve adherence.
8. Analysis of primary outcome	Intention-to-treat; may supplement with per-protocol analysis.		Intention-to-treat; pre-specified analyses with adequate power to detect heterogeneity of treatment effects within clinically important patient subgroups and practice settings.

Asthma offers numerous opportunities for CER. Over the last several decades, there has been substantial progress in the development of pharmacologic interventions addressing specific mechanisms of disease (excessive airway smooth muscle tone, airway inflammation, and IgE-mediated immune responses). There has also been demonstration of the efficacy of non-pharmacologic interventions, ranging from asthma self-management strategies, practice innovations to achieve best practices, to bronchial thermoplasty. To improve the health of patients with asthma, evidence-based treatment guidelines have been developed to provide patients and providers a synthesis of the evidence base for best practices.^20,21 However, only a small minority of the patients with asthma would meet the eligibility criteria for the major trials on which such treatment guidelines are based.¹ Thus, CER studies are needed to establish the relative benefits vs. harms of different treatment strategies in populations more representative of actual practice.

Of particular importance are the approximately 20% of patients with asthma who are smokers. Smoking impairs responses to inhaled and systemic corticosteroids,^22–24 which are guideline-recommended as first-line medications for long-term asthma control and the treatment of exacerbations, respectively. Unfortunately, most major clinical trials in asthma exclude current and past smokers with greater than 5–10 pack-years. Patients with asthma have also been excluded from studies of chronic obstructive pulmonary disease (COPD) because they are not felt to be old enough or have had sufficient tobacco exposure to have COPD. This situation has resulted in a paucity of data regarding effective treatments for asthma in a large subgroup of patients who have a greater symptom burden, worse lung function, and excess asthma morbidity (e.g., hospitalizations).^25–28 Another group deserving an emphasis in CER are older adults with asthma, since they account for a growing proportion of hospitalizations for asthma and have the highest rates of asthma-related deaths.^29,30 Unfortunately, older adults (e.g., older than 55 years, 65 years, or more), have also been traditionally excluded from major clinical trials in asthma. Largely based on the evidence available from efficacy studies, guidelines in asthma provide age-specific (e.g., 0–4 years, 5–11 years, and >12 years) recommendations for care. CER studies in asthma are needed to evaluate the ‘real-world’ effectiveness of care within and across these age strata, as well as the need for a stratum for older adults.

CER studies are also needed to define the most effective treatment strategy in other populations with excess asthma morbidity (e.g., racial/ethnic minorities, women, those with comorbid conditions, including mental health disorders) and those with specific asthma phenotypes.³¹ Because asthma care is delivered by a range of clinicians (specialists and non-specialist physicians; physician extenders) and across a range of practice settings (e.g., academic and non-academic healthcare practices; rural and urban), CER studies should ideally encourage participation of a broad range of clinicians and practice settings. Only then will we understand what works in the ‘real-world’.

Understanding treatment heterogeneity will necessitate decisions about how best to develop and fund studies of sufficient size to address subgroup specific questions prospectively and how best to prioritize such questions, as it is unlikely that a single trial can be sufficiently large to address all subgroups of interest simultaneously. Carefully designed CER studies that explicitly address treatment heterogeneity will provide exciting opportunities to develop an evidence-base for guiding decisions about how best to tailor care for individual patients with asthma, thereby providing the basis to bring together the goals of CER and personalized medicine.³²

While there is potentially an endless list of research questions that could be evaluated using a CER framework, it is unlikely that all such questions are of equal merit or urgency. Prioritization of CER studies in asthma is needed to ensure that the most pressing questions are addressed first. Traditionally, research prioritization has been the domain of funding agencies, which have relied on internal and external advisors (e.g., Advisory Councils) to identify areas in need of growth. Funding agencies have employed targeted and time-limited grant opportunities, such as Request for Applications (RFAs) to promote research in these areas. Research investigators have also been provided the opportunity to develop their own research priorities and submit grant applications dictated by the state-of-the-science to pursue questions arising from previous work.

We propose greater involvement of stakeholder groups in the prioritization of CER studies, including patients (and patient advocacy groups), clinicians (physicians, nurses, respiratory therapists, and other clinicians; specialists and generalists from diverse healthcare practice settings), insurers, regulators, quality improvement organizations, researchers, and funding agencies. An infrastructure that promotes collaboration between the various groups that develop, disseminate, and use new knowledge, is, in our view, an essential component of CER. Such collaboration will ensure that a diverse set of perspectives contribute to the identification of research topics and that CER provides the information needed by end users. Such a collaborative infrastructure will also facilitate dissemination and implementation of study results once available and therefore narrow the research to practice gap that characterizes much of healthcare for asthma today. The COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) was funded by AHRQ to bring together a diverse group of stakeholders in COPD to develop and prioritize an effectiveness / implementation research agenda for COPD.³³ In collaboration with stakeholders and with NHLBI funding, the CONCERT consortium is now developing a national data infrastructure to conduct COPD CER studies that address this research agenda. We suspect similar opportunities exist for asthma.

In summary, the goal of CER in asthma is to provide actionable evidence about the relative harms and benefits of existing pharmacologic and non-pharmacologic options for the care of individual patients and how best to deliver this care in real-world settings. Key challenges include defining the relative roles of observational CER and effectiveness trials in providing this evidence. Because trials can fall along a multi-dimensional continuum of efficacy to effectiveness, there also needs to be consensus on the minimum criteria for defining an effectiveness trial --in other words, how far to the right on the efficacy to effectiveness continuum is sufficient? Other areas that deserve discussion include the development of robust analytic approaches that are specific to asthma to minimize the risk of confounding and selection bias in observational CER studies, the role of pre-post interventions and other designs as alternatives to randomization in effectiveness trials, the promotion of cross-study synthesis of an effectiveness evidence base by identifying ‘core’ patient-centered asthma outcomes, and the development multi-center networks to design and conduct observational CER studies and effectiveness trials that help translate current investments in basic research and efficacy research to improved asthma healthcare and outcomes. We recommend broad stakeholder input in these discussions, as well as in developing and in prioritizing a research agenda for asthma CER.

The time is here for patients with asthma to receive the most appropriate care, for clinicians to deliver personalized medicine, and for researchers to develop the evidence-base that helps more patients and clinicians at the point-of-care. We call for action on CER in asthma and look forward to the exciting times ahead.

LIST OF ABBREVIATIONS

AHRQ

Agency for Healthcare Research and Quality

CER

Comparative Effectiveness Research

CONCERT

COPD Outcomes-based Network for Clinical Effectiveness and Research Translation

COPD

Chronic obstructive pulmonary disease

NHLBI

National Heart, Lung, and Blood Institute

PCORI

Patient-Centered Outcomes Research Institute

PRECIS

Pragmatic-Explanatory Continuum Indicator Summary