Clinical Comparative Effectiveness Research Through the Lens of Healthcare Decisionmakers

Abstract

Background

Healthcare expenditures in the United States exceed the healthcare expenditures of other countries, yet relatively unfavorable health outcomes persist. Despite the emergence of numerous evidence-based interventions, wide variations in clinical care have caused disparities in quality of care and cost. Comparative effectiveness and cost effectiveness research may better guide healthcare decisionmakers in determining which interventions work best, for which populations, under which conditions, and at what cost.

Methods

This article reviews national health policies that promote comparative effectiveness research (CER), healthcare decisionmaker roles in CER, methodological approaches to CER, and future implications of CER.

Results

This article provides a brief summary of CER health policy up to the Patient Protection and Affordable Care Act and its establishment of the Patient-Centered Outcomes Research Institute (PCORI). Through PCORI, participatory methods for engaging healthcare decisionmakers in the entire CER process have gained momentum as a strategy for improving the relevance of research and expediting the translation of research into practice. Well-designed, methodologically rigorous observational studies and randomized trials conducted in real-world settings have the potential to improve the quality, generalizability, and transferability of study findings.

Conclusion

Learning health systems and practice-based research networks provide the infrastructure for advancing CER methods, generating local solutions to high-quality cost-effective care, and transitioning research into implementation and dissemination science—all of which will ultimately guide health policy on clinical care, payment for care, and population health.

INTRODUCTION

Per capita healthcare spending in the United States continues to be among the highest in the world; however, this investment has not translated into better health outcomes compared to other high-income countries.¹ Wide variations in treatments, outcomes, and costs clearly indicate a need for improvement in the US healthcare system. These problems are fueling demands from healthcare decisionmakers for more evidence of the comparative effectiveness and cost effectiveness of medical interventions.

Comparative effectiveness research (CER) is believed to be the mechanism that will fill current knowledge gaps in healthcare decisionmaking.^2-3 The Institute of Medicine (IOM) National Priorities Committee defines CER as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care.”² The purpose of CER is to “assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve healthcare at both the individual and population levels.”² The IOM further emphasizes the need to directly compare alternative interventions, study patients in real-world clinical settings, and strive to tailor medical decisions to individual (or subgroup) values and preferences.

Given the increased prominence of CER for healthcare decisionmaking, this article provides a brief overview of the (1) historic context of national health policies supporting CER; (2) role of healthcare stakeholder engagement in CER; (3) methodological considerations in the conduct of CER; and (4) future implications for research, clinical practice, and health policy.

LITERATURE SEARCH

The author searched PubMed for methodological guidelines and publishing standards for CER using the following search strategy: (comparative effectiveness research [MeSH Terms] OR (comparative [All Fields] AND effectiveness [All Fields] AND research [All Fields]) OR comparative effectiveness research [All Fields]). The search, limited to review articles published between 2009 and 2014, yielded 941 articles. The search was further restricted to articles with the search terms in the title, yielding 20 articles. From these articles, the author selected general reviews on research methodology and standards for reporting and then searched PubMed for related articles. The author also searched the reference lists of articles identified as key references to find additional articles. The author employed the same search strategy to identify key reference articles that discussed CER within the context of the Patient Protection and Affordable Care Act (PPACA) and stakeholder engagement. Table 1 provides an overview of the key findings of this review process.

Table 1.

Summary of Comparative Effectiveness Research

HEALTH POLICIES

After passage of the Social Security Amendments of 1965 that established Medicare and Medicaid, health services research became especially important to Congress as the members of the House of Representatives and Senate struggled to figure out how to contain rising healthcare expenditures.⁴ In 1999, Congress established the Agency for Healthcare Research and Quality (AHRQ) that was tasked with examining (1) outcomes, effectiveness, and cost effectiveness of medical practices and technologies; (2) utilization and access to care; (3) organization, delivery, and financing of services and their interaction with and impact on quality of care; (4) methods for measuring and strategies for improving quality of care; (5) strategies for engaging patients in their care; and (6) methods by which healthcare stakeholders learn best practices and use this information for healthcare delivery.^5,6 In 2003, the Medicare Prescription Drug, Improvement, and Modernization Act created the Effective Health Care Program to expand AHRQ's responsibility to include CER.⁴ In 2009, the American Recovery and Reinvestment Act appropriated $1.1 billion for CER and tasked the IOM with establishing research priorities. The IOM subsequently recommended 100 CER priorities—many of which focus on the need to improve health service delivery.²

The PPACA moved the United States toward a national policy for CER to increase accountability for quality and cost of care. The PPACA established the Patient-Centered Outcomes Research Institute (PCORI) as a government-sponsored nonprofit organization to advance the quality and relevance of clinical evidence that patients, clinicians, health insurers, and policy makers can use to make informed decisions.⁴ PCORI's funding source is the Patient-Centered Outcomes Research Trust Fund that receives funding from the Federal Hospital Insurance Trust Fund, the Federal Supplementary Medical Insurance Trust Fund, the Treasury general fund, and fees on health plans to support CER.⁷ The PPACA defines CER as “research evaluation and comparing health outcomes and clinical effectiveness, risks, and benefits of two or more medical treatments, services, and items.”^4,7 The PPACA defines treatment, services, and items as “healthcare interventions; protocols for treatment, care management, and delivery; procedures; medical devices; diagnostic tools; pharmaceuticals; integrative health practices; any other strategies or items being used in the treatment, management, and diagnosis, or prevention of, illness or injury in individuals.”^4,7 The law further specifies that PCORI must ensure that CER accounts for differences in key subpopulations (eg, race/ethnicity, gender, age, and comorbidity) to increase the relevance of the research.

PCORI formally incorporated the concept of “patient-centeredness” into CER and characterized patient-centered outcomes research (PCOR) as (1) comparing alternative approaches to clinical management, (2) actively engaging patients and key stakeholders throughout the research process, (3) assessing outcomes that are meaningful to patients, and (4) implementing research findings in clinical settings.⁸ Examining the impact of interventions on patient-reported outcome measures such as symptom severity, functional status, and quality of life is an imperative component of PCOR. The best way to determine which outcomes matter most to patients and their caregivers is to engage them in the research process.

STAKEHOLDER ENGAGEMENT

Engaging stakeholders in research improves the relevance of study questions, increases transparency, enhances study implementation, and accelerates the adoption of research findings into practice and health policy.⁹ The degree of stakeholder participation depends on interest, expertise, negotiation, and/or project governance structure. Stakeholders are categorized into 7 groups, including patients and the public, providers (individuals or organizations), purchasers (responsible for underwriting costs of care), payers (responsible for reimbursement), policy makers, product makers (drug/device manufacturers), and principal investigators (researchers or their funders).¹⁰ Two 2014 reviews of stakeholder engagement in CER and PCOR demonstrate that patients are the most frequently engaged stakeholder group, engagement most often occurs in the early stages of research (prioritization), and stakeholder roles in research are highly variable.^10,11 Engagement strategies range from surveys, focus groups, and interviews to participation in study advisory boards or research teams. No clear evidence supports any particular strategy for engaging patient stakeholders as better than others.¹¹

The PCORI Patient and Family Engagement Rubric describes stakeholder engagement in the study planning, study implementation, and dissemination of results for CER and PCOR.¹² The rubric is not intended to be comprehensive or prescriptive. However, it outlines 4 PCOR engagement principles: (1) reciprocal relationships (clearly outlining the roles of all research partners, including patients); (2) colearning (a bidirectional process in which patient partners understand the research process and researchers understand the principles of patient-centeredness and engagement); (3) partnership (fair financial compensation, thoughtful consideration for the time commitment requested, and accommodation for cultural diversity); and (4) trust/transparency/honesty (inclusive decisionmaking, sharing information with all partners, commitment to open and honest communication, and communicating study findings in meaningful and usable ways).¹²

METHODOLOGICAL CONSIDERATIONS

Study Designs

Although CER and PCOR utilize participatory approaches to research, stakeholder engagement does not preclude the need to employ the methodological rigor of research. Careful attention to study design is imperative. The principal methods for CER are observational studies (prospective and retrospective), randomized trials, decision analysis, and systematic reviews.¹³ Table 2 summarizes 3 studies that employ the 2 main forms of CER—randomized trials and observational studies. The Adherence and Intensification of Medications (AIM) study demonstrates the value of conducting randomized controlled pragmatic trials.¹⁴ The Swedish Obese Subjects (SOS) trial is one of the largest prospective observational cohort studies to date.¹⁵ The Initial Choice of Glucose-Lowering Medication for Diabetes Mellitus study used insurance claims data for a retrospective cohort study.¹⁶

Table 2.

Examples of Comparative Effectiveness Research (CER) Using Pragmatic Clinical Trial and Observational Study Designs

The advantage of observational studies is that they can quickly provide low-cost, large study populations. Observational studies also include data from diverse patients obtained during routine clinical practice that strengthen the external generalizability of study findings. Nonetheless, observational studies are limited by the inherent bias and confounding of results that routinely occur in nonrandomized studies.^13,17,18 To minimize the threats to the internal validity of observational studies, research guidelines recommend the following: a priori specification of research questions, targeted patient populations, comparative interventions, and postulated confounders; selection of study designs that are appropriate to the study questions; selection of the appropriate data source; and transparency in protocol development and prespecified analytic plans.^18,19 A discussion of analytic methods (regression analysis, propensity scores, sensitivity analysis, instrumental variables, and structural model equations) for observational studies is beyond the scope of this article, but these methods are discussed in detail in several reviews of CER.^13,17,18

The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Good Research Practices Task Force provides detailed recommendations on how to determine when to do a prospective vs retrospective study, the advantages and disadvantages of different study designs, and analytic approaches to consider in study execution.^18,19 Several designs have been developed for prospective observational studies.¹⁸ The single group, pretest/posttest design is a longitudinal study in which subjects serve as their own control, and outcomes are collected before and after an intervention. In contrast, the multiple group, pretest/posttest design collects outcomes for at least 2 comparison groups. The multiple group, cross-sectional design involves study participants with a particular condition who have already undergone one of multiple interventions. Prospective cohort studies are longitudinal studies in which outcomes are only collected after an intervention.

The ISPOR task force also provides guidelines for conducting retrospective observational studies on secondary data sources (eg, claims databases and electronic medical records).¹⁹ Electronic databases contain information collected for operational reasons rather than research purposes and therefore have minimal reporting bias. However, data quality (eg, missing/incomplete data), selection bias, and unmeasured confounding inherent to data collected in clinical practice are major threats to internal validity. Careful attention to the epidemiologic study design (cross-sectional, cohort, case-control, case-crossover) and statistical methods is critical to enhancing study validity. Cross-sectional studies provide a snapshot of data but limit the ability to establish the temporality of exposure to an intervention relative to the outcome of interest. When temporality of exposure is of particular interest, cohort studies are ideal. Case-control designs are historically helpful when the outcome of interest is rare. The case-crossover design, in which individuals serve as their own control, is ideal when transient exposures result in acute events or outcomes. Retrospective studies, if done well, can supplement evidence from prospective observational studies and randomized trials.

While the more costly traditional randomized controlled trials (RCTs) have the advantage of strong internal validity, their restrictive inclusion criteria tend to result in homogenous groups of study participants that are not reflective of what clinicians see in real practice. For cost reasons, RCTs are often limited in size (limiting the ability to detect adverse effects) and in duration (limiting the ability to observe long-term outcomes).

RCTs require large populations to implement the key elements of CER as defined by the IOM (comparison of alternative interventions in real-world settings and tailored to the values of individuals or subpopulations) and to avoid false conclusions.¹³ Cluster randomized trials meet the CER criteria of direct comparison of interventions because randomization occurs at the practice level instead of the individual patient level. Implementation of a single intervention at a site resembles what happens in real practice. Pragmatic or practical trials examine interventions that are currently in use in typical practice settings and include patients with demographic profiles that are similar to patients routinely treated in real practice.^3,13,20,21 The major drawback to pragmatic trials is that they usually require larger sample sizes and longer follow-ups of major clinical outcomes compared to traditional RCTs to better reflect the natural history of disease. To minimize costs, pragmatic trials may limit outcome measurements to easily obtained data.¹³ Doing so, however, may result in not collecting cause-specific measures, thus limiting the ability to ascertain why an intervention is or is not effective. Adaptive trials change design in response to prespecified criteria and study data collected that may reveal early indications of a study's ultimate outcomes.¹³ The changes can occur in any aspect of the study—the number of arms, types of intervention, sample sizes, sampling strategy for subgroups of interest, or outcome measures. Adaptive changes maximize study efficiency and increase relevance.

Regardless of which study design is employed (observational or randomized trial), transparency and adherence to methodological standards will enhance generalizability and transferability of study results across populations, settings, and systems of care. A number of research guidelines are available to help investigators appropriately plan and execute methodologically rigorous CER studies.^{13,17-19,22-28}

Integrating Research in Clinical Practice

To expand CER, researchers must have conducive practice environments in which to conduct studies. Learning health systems and practice-based research networks may be uniquely positioned to meet the infrastructure needs of CER because they are positioned to promote research prioritization, evidence generation, and translation of evidence into practice.^29-31

In learning health systems, research and clinical practice are tightly integrated; thus research priorities are aligned with key issues clinicians face in everyday practice, and research on those issues informs best practice.²¹ Key attributes of learning health systems include proactively identifying problems to guide research priorities, testing pilot interventions to identify strategies for successfully implementing interventions in diverse settings, evaluating interventions with predefined impact measures, adjusting interventions to the contextual environment, and disseminating findings internally and externally.³² Learning health systems such as hospital-based CER centers may be the ideal model for improving evidence-based practice and cost containment.³³ Hospital-based CER can harness local data on utilization, outcomes, and cost of care from electronic medical records and other data warehouses to identify gaps in service or practice. To close the gap, clinical decision support and quality improvement initiatives can be integrated into the health system while using administrative and clinical data to monitor performance.

Practice-based research networks (PBRNs) are organized networks of ambulatory practices involved in primary care research. According to the AHRQ, “PBRNs draw on the experience and insight of practicing clinicians to identify and frame research questions whose answers can improve the practice of primary care. By linking these questions with rigorous research methods, the PBRN can produce research findings that are immediately relevant to the clinician and, in theory, more easily assimilated into everyday practice.”³⁴ As of July 2012, the AHRQ reported that of the 136 PBRNs in the United States, 15% were national and 28% were regional network collaborations.³⁵ Greater external generalizability of study results from these large networks is the obvious advantage. PBRNs are especially attractive for recruiting priority patient populations that are often underrepresented in clinical research (eg, multiple chronic conditions) and for collecting comprehensive data on patient-reported outcomes, practice settings, and contextual factors that affect healthcare decisionmaking.³¹ Research in PBRNs is usually aligned with quality improvement activities increasingly using participatory methods to not only make the research relevant but also to improve the translation of research into practice.³⁶ PBRNs are rapidly evolving into learning collaboratives, given the nature of the research and various engagement strategies.

FUTURE IMPLICATIONS

Improving the quality and relevance of evidence on the clinical effectiveness of healthcare services holds promise for shaping policies related to clinical care, payment for services, and population health. Increased emphasis on implementation science to accelerate the use of evidence-based interventions in clinical practice is needed.³⁷ Participatory research methods are likely to produce evidence-based interventions that are more relevant and actionable to healthcare decisionmakers compared to nonparticipatory research. Healthcare organizations can advance CER by investing in (1) personnel dedicated to data collection, monitoring, and interpretation; (2) data infrastructure and analytics; (3) evidence-based quality improvement initiatives; and (4) adaptive implementation and dissemination strategies.³⁸ A systems approach to implementing evidence-based interventions in research is necessary to account for the dynamic, complex environments in which patient care occurs.³⁷

CONCLUSION

Broadening approaches to research whereby investigators use study designs that best match research questions rather than limiting discovery research to randomized trials is critical to advancing CER. Prospective, practice-based studies will help clarify the resources needed to implement evidence-based interventions and identify which interventions work best for specific populations and settings. Given the connection between local healthcare delivery and national healthcare expenditures, any local knowledge gained from CER in learning health systems and PBRNs automatically has implications for national health policy.

This article meets the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties Maintenance of Certification competencies for Medical Knowledge and Systems-Based Practice.