Comparative Effectiveness Research in Lung Diseases and Sleep Disorders

Abstract

The Division of Lung Diseases of the National Heart, Lung, and Blood Institute (NHLBI) held a workshop to develop recommendations on topics, methodologies, and resources for comparative effectiveness research (CER) that will guide clinical decision making about available treatment options for lung diseases and sleep disorders. A multidisciplinary group of experts with experience in efficacy, effectiveness, implementation, and economic research identified (a) what types of studies the domain of CER in lung diseases and sleep disorders should include, (b) the criteria and process for setting priorities, and (c) current resources for and barriers to CER in lung diseases. Key recommendations were to (1) increase efforts to engage stakeholders in developing CER questions and study designs; (2) invest in further development of databases and other infrastructure, including efficient methods for data sharing; (3) make full use of a broad range of study designs; (4) increase the appropriate use of observational designs and the support of methodologic research; (5) ensure that committees that review CER grant applications include persons with appropriate perspective and expertise; and (6) further develop the workforce for CER by supporting training opportunities that focus on the methodologic and practical skills needed.

Recent congressionally mandated federal efforts to establish a “robust comparative effectiveness enterprise” have generated definitions, priorities, and organizational mechanisms for conducting comparative effectiveness research (CER) (1–3). The National Institute of Health's commitment to CER is reflected in its history of comparative clinical trials that have profoundly influenced clinical practice as well as its current involvement in shaping the CER enterprise (4).

The NIH currently uses the Federal Coordinating Council definition of CER (5, 6): “…the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in ‘real world settings’” (5). Operationalizing this definition to shape a specific research agenda poses numerous challenges, including defining where CER would best fill the gaps in clinical evidence for particular lung diseases or sleep disorders, understanding how to use innovative study designs, obtaining large sample sizes and data sets to examine subgroup variations in responses, using electronic health records and integrated data systems to increase efficiency, and identifying the role for observational studies.

The Division of Lung Diseases (DLD), National Heart, Lung, and Blood Institute (NHLBI) convened a workshop in September 2010 to discuss the role of CER in DLD-supported research and to make recommendations to help assure that CER research in lung diseases and sleep disorders meets the challenges and opportunities noted above. Participants included clinical investigators in asthma, chronic obstructive pulmonary disease (COPD), acute respiratory distress syndrome (ARDS), pediatric lung diseases, and sleep disorders, as well as methodologists with expertise relevant to CER.

Before the workshop, participants completed a survey asking about priority topics for CER lung diseases overall and in their respective fields, the barriers to conducting CER, and the most important steps to advance CER in their fields. Results were used to provide a framework for the meeting agenda. At the workshop, participants were divided into subgroups to design a mock CER intervention trial and a mock CER observational study. This activity provided a practical consideration of the themes that emerged from the morning's more general discussion of CER in lung disease. Group discussion of the mock trials and the key issues and concerns that emerged in their development led to the specific recommendations to advance CER in lung diseases presented in this report.

The Domain of CER in Lung Diseases

Efficacy Versus Effectiveness

The efficacy of a treatment refers to its ability to improve patient outcomes in a population likely to respond to the treatment, with strict adherence to the treatment protocol, in specialized centers and with highly motivated providers. An efficacy experiment, most often a randomized controlled trial (RCT), answers the question, “Can this intervention achieve a beneficial effect under highly favorable circumstances?” Generally, efficacy research precedes effectiveness research. The effectiveness of a treatment refers to its ability to improve patient outcomes in more real-world settings, for example, with a more heterogeneous population exhibiting the normal spectrum of disease, comorbid illness, ethnicity, and/or age, with average compliance with treatment protocols and medications, and/or in a wider spectrum of practice settings. Effectiveness research, even with large study populations in community settings, often includes randomized controlled study designs.

What Is CER?

CER is within the continuum of effectiveness research. CER is characterized by the study of two or more interventions using a variety of study designs, which allows comparative evaluation of the effectiveness of these interventions in real-world settings, for diverse patient populations and subgroups, assessing a full range of outcomes to address questions important to patients, caregivers, and other key stakeholders. Examples of such outcomes include clinical events, measures of disease control, symptom status, activity limitations, work or school absenteeism or lost productivity (presenteeism), and disease impact or burden. There are a number of independent measures of these outcomes that are not limited to quality-of-life questionnaires. Not all effectiveness research is CER. For example, an RCT comparing a medication or other therapy to placebo, even if conducted in large populations under real-life conditions, would be an effectiveness study but would not meet criteria for CER.

The NIH working definition of comparative effectiveness research is as follows: CER is the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in “real-world” settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.

• To provide this information, comparative effectiveness research must assess a comprehensive array of health-related outcomes for diverse patient populations and subgroups.

• Defined interventions compared may include medications, procedures, medical and assistive devices and technologies, diagnostic testing, behavioral change, and delivery system strategies.

• This research necessitates the development, expansion, and use of a variety of data sources and methods to assess comparative effectiveness and actively disseminate the results (5).

The workshop participants discussed this definition and identified the following core elements to emphasize for CER in lung diseases and sleep disorders: (1) comparison of interventions or implementation strategies in real-world settings, (2) assessment of a comprehensive array of health-related outcomes, (3) studies with diverse patient populations and attention to important subgroups, (4) use of a variety of observational or experimental methods.

Implementation research—the scientific study of how to promote the adoption of research findings (7)—should be supported to assure that effective interventions are adopted into practice without needless delays. For example, some implementation studies have evaluated interventions directed toward providers to improve clinical practices, such as audit and feedback, local opinion leaders, and educational outreach visits (8–10). Implementation research may also evaluate the effects of changes in health care systems or financing on translating research into practice. Workshop participants also recommended that costs should be considered using rigorous economic evaluation methodologies.

The workshop participants were concerned that the NIH emphasizes comparison of interventions of proven efficacy. However, there will be many CER questions about lung diseases and sleep disorders that may be best answered by comparing a proven intervention to usual care, and workshop participants felt the NIH definition should be interpreted to include this possible design as long as there is appropriate equipoise between the proven intervention and the active comparator of usual care. To include diverse patient populations, a number of different approaches may be needed, depending on the specific research questions addressed. The NIH definition's recommendation to use a variety of data sources and methods suggests support of, for example, innovative use of practice-based research networks, registries and pooled data from electronic medical records, and appropriate use of prospective and retrospective observational study designs.

CER Study Designs

CER can be conducted using interventional or observational study designs (Table 1). Arguably, randomized studies provide the strongest evidence about the effect of a treatment because they are less prone to bias than observational designs. They are not always feasible or affordable and usually require several years to produce results. Pragmatic trials (also called practical trials, which are randomized trials designed for typical, real-world settings and may include relaxed inclusion/exclusion criteria, relaxed simple and/or flexible protocols, and outcome measures of relevance to patients, physicians, and payers) and group-randomized trials (trials in which the subjects are assigned to intervention or control in groups, or clusters, defined by a common features) (11) are increasingly used approaches to increasing the size and efficiency of randomized designs (12, 13). Pragmatic trials achieve better generalizability than efficacy trials, at some cost to internal validity (12). Group-randomized trials are especially useful when individual-level randomization is not possible, and they are well suited to studies involving health care systems or multisite networks of providers, schools, or communities. When randomization is not feasible, novel approaches to the design and analyses of interventions, such as interrupted time series analysis, can be used in the setting of intervention studies.

TABLE 1.

STUDY DESIGNS FOR COMPARATIVE EFFECTIVENESS RESEARCH

Study Design	Characteristics	Example	Advantages and Limitations
Experimental designs
Pragmatic trials	Large samples are recruited; may use multiple centers and/or community settings; have limited inclusion and exclusion criteria; address outcomes important to stakeholders	Does prenatal betamethasone reduce the risk of neonatal respiratory distress syndrome after elective caesarean section? This study enrolled 998 patients via 10 centers (40)	The results of these trials tend to be more generalizable and the design maintains the scientific rigor of a traditional RCT (11)
Cluster-randomized trials	Groups rather than individuals are randomized	Does computerized decision support enhance the management of asthma and angina in primary care? This study randomized 60 general practices (41)	These trials are useful for comparing alternative established therapies (11) and for comparing changes in health care systems
Quasi-experimental designs
Interrupted time series and regression discontinuity designs	Take advantage of designed interventions or natural experiments created by policy changes. Interrupted time series designs compare rates or trends in the outcome before and after the policy change/intervention period	Was the introduction of pneumococcal conjugate vaccination associated with reduced pneumonia hospitalizations among U.S. children? This study compared trends in pneumonia hospitalization before and after the vaccine's adoption (15)	These designs are less prone to confounding, but can only be conducted under selected circumstances.
Observational designs
Pre- and postintervention, with comparison group(s)	Compare outcomes in the pre- and postintervention periods between the intervention and comparison groups		This design is weaker than the interrupted time series and regression continuity designs because of concerns about selection and baseline comparability of the intervention and comparison groups
Pre- and postintervention without a comparison group	Compare outcomes between the pre- and postintervention periods in the intervention group alone		This is considered a much weaker design than the above two types
Prospective cohort	Compare groups assigned to different treatments in real-life settings; existing data may be used but are supplemented with primary data collection; useful for patient-reported outcomes	Is right heart catheterization associated with better clinical outcomes in critically ill patients? This study used a prospective cohort design with statistical techniques to adjust for confounding (16)	These studies can collect more detailed information than available for retrospective cohort studies; they are also prone to confounding. Their findings may suggest the need for a future RCT
Retrospective cohort	Compare groups assigned to different treatments in real-life settings; existing data are used	Are inhaled corticosteroids associated with lower risk of hospitalization for asthma (42)?	These studies take advantage of existing data but are prone to confounding. Their findings may suggest the need for a future RCT

Definition of abbreviation: RCT = randomized clinical trial.

Observational study designs are an important component of CER, and workshop participants endorsed increasing their use for lung diseases and sleep disorders. Observational designs can take advantage of natural experiments or existing variation in practice. They can be conducted retrospectively or prospectively. They often use existing computerized data; additional information can be collected via medical record review or surveys. Natural experiments may occur when there are changes in the availability of an intervention. In one example, Medicaid policies in one state were changed to limit access to psychotropic medications, enabling researchers to evaluate effects on health outcomes using interrupted time-series analysis (14). Conversely, a new treatment, such as pneumococcal conjugate vaccination of children, may become widely adopted at a specific point in time, affording investigators the chance to evaluate its effects (15).

Observational research is preferred when interventional studies are not feasible (for example, when randomization is not acceptable due to lack of equipoise) or when key outcomes are delayed and long-term follow-up would be impractical. Observational studies are more likely to allow timely comparisons in diverse populations and in real-world settings than randomized trials. In addition, they can contribute critical information to characterize actual practice in real-life settings and are very useful before the design of large RCTs to identify study questions that are likely to produce results that will change care. At times, they have helped create equipoise that allows RCTs to be performed (16).

An important limitation of observational study designs is the limited ability to infer causation. There are striking examples wherein multiple observational studies suggested one treatment but subsequent randomized studies showed opposite results (e.g., hormone replacement therapy for menopausal women). A related limitation is confounding by indication, in which treatment decisions are associated with patient characteristics that influence outcomes. Methods used to address confounding include propensity scores, instrumental variables, restriction, interrupted time-series designs, and regression discontinuity designs (17–23). However, propensity scores do not control for unidentified confounders, and instrumental variable analyses may have limitations in this regard.

The workshop participants agreed that grant applications for CER in lung diseases might choose among a variety of methods and should provide a clear justification why the method chosen is best for the question being addressed. Applicants may also propose to apply different methodological approaches to evaluate whether results are consistent across them. Workshop participants also endorsed further methodological research to identify optimal existing strategies and to develop new approaches to common issues, such as adjustment for confounding in observational CER.

Criteria and Process for Setting Priorities Among Topics for CER

Criteria will be needed to set priorities for CER in lung diseases and sleep disorders. Workshop participants noted that priorities need to be set within three general areas: (1) identification of specific lung diseases and sleep disorders and research questions within them, (2) methods for CER, and (3) infrastructure for CER. Workshop participants endorsed the criteria for setting priorities defined by the Federal Coordinating Council on CER and listed in the NIH definition of CER and added specific comments for consideration in lung disease CER (Table 2).

TABLE 2.

COMMENTS ON THE NATIONAL INSTITUTES OF HEALTH CRITERIA FOR DEFINING STUDIES AS COMPARATIVE EFFECTIVENESS RESEARCH AND FOR SETTING PRIORITIES AMONG COMPARATIVE EFFECTIVENESS RESEARCH STUDIES

Criteria	Comments by Workshop Participants
Minimum or threshold criteria for a study to be defined as CER
Inclusion within the statutory limits of the Recovery Act and the Council's definition of CER	Options for comparators include intervention(s) with known efficacy but limited information about effectiveness1 and usual care2
	1Shown to have favorable harm to benefit profile in research settings, but for which limited  or no evidence of effectiveness exists;
	2Range of medical practices found in clinical practice. Usual care vs. usual care plus active intervention provides the opportunity to understand incremental value of interventions.
Potential to inform decision making by patients, clinicians, or other stakeholders	CER should be conducted in real-world settings to enhance the applicability of results to clinical practices. Study participants should represent the spectrum of patients seen in clinical practice. Ideally, interventions should be conducted by typical clinicians, not researchers or research staff.
	Responsiveness to expressed needs of patients, clinicians, or other stakeholders	CER studies should explain how the needs of stakeholders were considered in the design, conduct, and dissemination of results of studies.
CER studies should reflect timely topics of clinical concern.
Feasibility of research topic (including time necessary for research)	Issues related to feasibility may also influence choice of study design for CER (observational, quasi-experimental, and experimental studies).
NIH criteria for prioritization
Potential impact (based on prevalence of condition, burden of disease, variability in outcomes, costs, potential for increased patient benefit or decreased harm)	The definition of benefit may depend on the perspective taken. For example, health care payers and providers have traditionally focused on clinical outcomes, whereas patients may focus more on functional status and other patient-centered outcomes.
Potential to evaluate comparative effectiveness in diverse populations and patient subgroups and engage communities in research	Profile of harms and benefits may vary across subgroups. Enrolling diverse populations provides the opportunity to identify heterogeneity of treatment effects and determine strategies for delivering personalized care.
Uncertainty within the clinical and public health communities regarding management decisions and variability in practice	Identifies research topics that have high potential to inform decision-making by patients, clinicians, or other stakeholders.
Addresses need or gap unlikely to be addressed by other organizations	Studies that address methodologic issues or studies unlikely to be sponsored by industry (e.g., active comparator studies or studies testing interventions with relatively small profit potential).
Potential for multiplicative effect (e.g., lays foundation for future CER, such as data infrastructure and methods development and training, or generates additional investment outside government)	Multiplicative effect may mean laying foundation for future efficacy studies or pragmatic clinical trials by conducting observational CER studies. This criterion also encourages development of new methods, which should be supported by NIH as well as AHRQ and other entities.

Definition of abbreviations: AHRQ = Agency for Healthcare Research and Quality; CER = comparative effectiveness research; NIH = National Institutes of Health.

CER is intended to address the expressed needs of end users, including patients (e.g., advocacy groups, community organizations), clinicians (e.g., professional organizations), health care providers (hospitals, health systems), and health care payers/policy makers (including insurers and the Centers for Medicare and Medicaid Services and the Veterans Administration). Examples of areas in which end-user involvement can enhance the CER enterprise, for both investigator-initiated and institute-prioritized research, include the prioritization of research questions, providing advice about the study design used to address the research question (e.g., selection of comparators, feasibility of conducting randomized or controlled clinical trials, quasi-experimental, or observational studies), and development of partnerships for data collection and for dissemination/implementation activities once study results are available. An active collaboration between those who develop, disseminate, and use the evidence generated from CER offers the greatest opportunity to provide actionable information to patients, clinicians, and other stakeholders, and therefore the basis to more rapidly translate CER findings into clinical practice.

Workshop experts recommended use of structured and transparent approaches to selecting priorities in CER. Various systematic methods for eliciting input from stakeholders have been developed, such as Analytic Hierarchy Process (24). Many methods have been applied primarily in business settings, but they could be adapted to apply to health research. Systematic approaches to setting research priorities offer an opportunity to understand the relative preferences of different stakeholder groups, as well as the criteria used by groups in prioritization. For example, the COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) researchers have developed a collaboration with a diverse group of stakeholders (patient advocacy groups, professional organizations representing physicians, nurses, respiratory therapists, and social workers, health plans/insurers, quality improvement organizations, and the Joint Commission) to develop and prioritize a research agenda for CER in COPD (25, 26) CONCERT has also engaged stakeholders to inform the planning and design of CER studies responsive to this research agenda (e.g., selection of populations, interventions, comparators, outcomes), providing the opportunity to develop an evidence base that directly meets the expressed needs of end users.

Barriers to CER in Lung Diseases and Resources to Overcome Barriers

Presently, there are several general impediments and some barriers specific to lung diseases and sleep disorders for the accomplishment of CER. The group identified these barriers, considered how they affect the ability to perform CER, and discussed potential strategies to overcome these obstacles.

Identification of Patients and Therapies for Studies

Many pulmonary disorders and some sleep disorders meet the criterion of a rare disease, defined by the NIH as a disorder that affects fewer than 200,000 people in the United States, which will present a challenge for CER. For example, cystic fibrosis (CF) is a chronic and debilitating disorder that often results in premature morbidity and mortality, but it only affects an estimated 30,000 individuals. In addition, regional variability exists in the prevalence of some pulmonary disorders, including interstitial lung diseases (27–29). The low prevalence and geographical heterogeneity of these diseases may hamper the ability to enroll these patients into pragmatic clinical trials.

The development of large national registries of patients with specific pulmonary diseases and sleep disorders might assist in the identification of patients for enrollment into clinical trials. Registries have generally been used for relatively rare diseases, such as CF. The information in the CF registry created more than 40 years ago by the CF Foundation allows caregivers and researchers to identify new health trends, design clinical trials for potential therapies, and identify patients for future trials. Few such registries currently exist for lung diseases and sleep disorders. To facilitate CER, it is necessary to develop more disease-specific registries, including developing methodologies and approaches for capturing key representative data on common complex pulmonary and sleep disorders.

Identification of large patient populations for CER in more common lung disorders can also be difficult, and systematic methods to aid recruitment are important. Clinical trial networks can help in identification of appropriate patients. The DLD has established a number of disease-specific networks that created the necessary infrastructure of clinical and associated support centers to enable the rapid development and conduct of multiple clinical protocols to evaluate the efficacy of promising diagnostic and therapeutic approaches. These clinical networks, which currently study asthma, interstitial lung disease, ARDS, and COPD, may be properly positioned to conduct CER if efforts are made to expand the patient pools and interventions outside of academic settings.

More recently, a large network of academic institutions supported by NIH Clinical and Translational Science Awards have led the development of a Web site for patients who are interested in participating in clinical research. The goal of this program, called ResearchMatch (30), is to link suitable patients with researchers who are looking for study participants. The NIH registry ClinicalTrials.gov is another resource to help connect patients with appropriate studies.

Another barrier to the performance of CER is the lack of proven therapies for many pulmonary and sleep diseases. For example, there is a paucity of proven medical therapies for ARDS and interstitial lung diseases. Large pragmatic trials that compare different therapies may not yet be practical for certain lung and sleep diseases. The identification of treatments with proven efficacy with smaller efficacy trials may be necessary before trials of the size and magnitude necessary for CER should be performed.

Electronic Data

Administrative data sets, including claims data files kept by health plans and government payers, are a potentially powerful tool for CER (31). Clinical information obtained during routine clinical care provides a unique opportunity to examine the relative effectiveness of treatments. The information derived from such records are coded data (International Classification of Diseases-9 codes), including diagnostic and procedure codes, dispensing records for medications, and free text entries. Newer technology, such as natural language processing, will be needed to take full advantage of the variety of different types of data available, such as free text that occurs in electronic medical records.

The Agency for Healthcare Research and Quality (AHRQ) has funded several large projects to build and expand the nation's clinical electronic infrastructure to generate prospective, patient-centered outcomes for CER on therapeutics and tests. The infrastructure could be scaled up to include other organizations and data systems, with the goal of sustainable support for CER. The development of new methods and data governance approaches needed to enhance the national infrastructure for CER will be spearheaded by the Electronic Data Methods Forum, a new initiative led for AHRQ by AcademyHealth. Descriptions of these AHRQ-sponsored projects are available on the program Web sites (32, 33).

Effectiveness studies that compare different mechanisms of health care delivery may also benefit directly from clinically derived data that can identify populations for study and/or allow assessment of the delivery and effectiveness of care. However, for investigators to make valid inferences from studies using such administrative data, they need to understand the validity of all the data elements to be used, including variables on exposures, outcomes, and covariates.

The quality of administrative data depends on limitations and changes in the specificity of the criteria for diagnoses and on temporal changes related to financial reimbursement and regional coding practices. Internation Classification of Diseases-9 codes require additional investigation to evaluate and improve the reliability and validity of these measures in heterogeneous clinical settings even for common conditions, such as COPD, asthma, sleep apnea, and acute lung injury. Furthermore, laboratory values require a reference standard to be meaningful, because results are known to vary across laboratories and time. Defining common reference standards is important for other elements of the medical record, including medications in which these elements may be described by name or coded according to other common standards, such as National Drug Classification or Systematized Nomenclature of Medicine codes. Harmonization of data records may need to occur not only across administrative data sets but also across electronic medical records (EMRs) from different settings. This will require transforming native data from disparate EMRs into data that have common meaning across information platforms. This issue is being addressed using standardized data dictionaries, such as those developed by the Health Maintenance Organization Research Network (34), the Mini-Sentinel Initiative sponsored by the Food and Drug Administration (35), and the COPD DataHub, CONCERT (26). For example, research in ontologies also may improve data integration efforts. Recent informatics research for sleep disorders research has led to the development of ontologic-driven data-mapping procedures that have provided an efficient approach for integrating data across databases and institutions (R. Mueller and colleagues, unpublished results).

A related concept is that certain CER questions may be best addressed by merging data from existing administrative or clinical databases (including EMRs) with data from prospectively collected research databases. Research databases often better standardize the definitions of data elements than do administrative or clinical databases. However, the formats of such databases may vary greatly and the coding systems may not easily harmonize between databases. Continued work on the development of standardized coding systems and ontological frameworks may facilitate the harmonization of data across many sources.

Although there are a number of questions regarding the validity and reliability of administrative data, another major barrier is access to the data. To understand the comparative effectiveness of care, up-to-date near real-time data are essential and becoming more common. Integrating data from multiple sources, including administrative data sets and EMRs, for CER purposes would significantly improve the ability to compare treatment and diagnostic strategies across multiple systems of health care. Because issues of privacy are determined by local institutional review boards (IRBs), there is significant heterogeneity in the willingness to permit sharing of data. There is currently no national standard for sharing data between health care providers, health care organizations, academic and nonacademic institutions, and federal entities. In research that relies on the sharing and pooling of data for individual use or reuse between such organizations, the process can be exceedingly inefficient and time consuming, with approvals required from multiple individuals within each organization, including IRB officials and security and privacy compliance officers. Despite having previously shared data between organizations, the process often must be replicated for each additional investigation, adding to the inefficiency and delaying research progress. Given the need to be able to perform comparative effectiveness research in an efficient and timely fashion, a national emphasis is needed to reduce this barrier. Emphasis needs to be placed on developing national standards to facilitate the sharing of data by agencies such as Centers for Medicare and Medicaid Services (CMS), NIH, and Department of Veterans Affairs and among academic and nonacademic partners. Such data sharing could involve deidentified or limited data sets in which personal identifiers such as names, birth dates, and dates of medical services are removed or transformed to protect patient privacy. These standards need to emphasize the relative benefit of CER in relationship to risk of disclosure associated with the use of clinically derived administrative data. Innovative strategies include distributed data network approaches wherein identifiable data elements remain under the control of the respective institutions and their disclosure or data sharing policies (36, 37). With appropriate governance arrangements, permissions for disclosure, access, and use of these data can be controlled by the originating health care system.

Funding Mechanisms

Investigator-initiated funding for CER is proportionately and in absolute terms far less than it is for other biomedical research. The need for CER to inform the nation's health care enterprise has motivated the use of targeted contracts, task orders, and other government-specified work. AHRQ has developed centers and networks, including Evidence-Based Practice Centers, Centers for Education and Research on Therapeutics, and the Developing Evidence to Inform Decisions about Effectiveness Network (38). These types of structures are important. In addition, the workshop participants emphasized the need for investigator-initiated peer-reviewed CER research via NIH mechanisms. Without funding to support innovative, peer-reviewed CER studies, the scientific quality of CER could suffer.

CER will benefit from multidisciplinary teams. These are not common and often do not include expertise in implementation or improvement research. In addition to the contract-based work done at AHRQ CER centers, programs and centers should be funded that assemble multiple disciplines and conduct investigator-initiated studies. Networks of investigators that have been established based on clinical trials, clinical and translational science awards (CTSA), or linkages of electronic data sets will be valuable resources. Approaches should be developed to encourage collaboration from investigators currently outside these networks.

Stakeholder Contributions

The ultimate impact of CER will depend on its implementation of treatment and strategies in real-world settings by the health care industry. Partnerships with stakeholders are needed, both to conduct research and to translate CER findings into practice. Although AHRQ, NIH, and the new Patient-Centered Outcomes Research Institute presumably will be the major funders of CER, other stakeholders will be needed to provide other resources crucial for CER implementation, including access to health care systems, providers, and patients; electronic data; and medical records. For example, health plans and provider groups might be able to provide access to claims and electronic medical records, pharmaceutical companies might conduct and provide clinical trials using novel designs, and information technology vendors or consultants could provide and develop data collection and interface methods to support CER. In addition, these partners could also provide settings and opportunities for training CER researchers. Potential barriers to these collaborations include economic interests that could be affected by the results and implementation of CER, cultural and political differences among the research community and stakeholders, organizational hurdles including legal concerns about patient privacy, and the lack of channels for communication and collaboration.

The National CTSA Consortium sponsored a CER forum in December 2010 at the NIH so that representatives from federal agencies, industry, foundations, and other stakeholders could discuss such high-impact CER topics as infrastructure, education/training/workforce development, methods development, community and practice engagement, and health information technology. The meeting outcomes included identifying challenges, opportunities, and next steps, including a recommendation that priority should be given to funding studies that have explicit plans to work with other stakeholders in the health care community. More information can be found at http://www.ctsaweb.org/.

Engagement of Community Participants

The CER reports by the Institute of Medicine (2) and Federal Coordinating Council for Comparative Effectiveness Research (5) emphasized the need for community input when choosing relevant questions and for recruiting sites, patients, and physicians into studies. These efforts also should address disparities in health and health care among communities. Engagement of the community has not been the tradition in academic medical research. This is an important part of CER and is mandated for work done at the AHRQ CER centers. Not only will such engagement enhance the relevance of questions addressed by CER but also it will engage members of the community in the research and will educate the public so it can maximally benefit from the findings of CER. Community engagement is a substantially new approach, there is relatively little experience in this area, and it is time consuming. Although expertise is sparse, it is growing through the efforts of Centers for Disease Control (CDC), NIH-supported community-based participatory research grants, and the Community Engagement Components of the NIH CTSAs. Support for developing the procedures and practices for engaging the public in CER is needed to further leverage these and encourage research in a wide variety of settings.

Needs for Training

The competencies needed by a researcher in CER include some that are distinct. Currently there is a critical need to strengthen a national cadre of investigators with the skills to conduct CER (39). Research training is a central role of CTSAs and they are involved in CER-specific education and career development. Investigators who conduct CER benefit from training in specific fields, especially epidemiology and biostatistics. More specifically, a recent report of the CTSA Consortium CER Workforce Development workgroup identified special skills and education relevant to CER, including research ethics and logistics related to the complex issues involved in community-engaged research, pragmatic clinical trials, biomedical informatics, electronic health records research, large database research, practice-based network research, and decision analysis/cognitive sciences, health economics/cost-effectiveness, and health services research (39). CER training and career development should be supported by training grants. Investigators who conduct CER may have Master in Public Health degrees, PhDs, or similar training.

Research institutions as well as investigators may view CER as representing a large paradigm shift. Research-intensive academic centers may be most familiar with traditional efficacy studies and randomized trials, and may not understand or value the role CER has in the spectrum of clinical research. IRBs may be uncomfortable with the flexibility needed in pragmatic clinical trials or with alternative approaches to seeking informed consent from patients in real-world settings. Thus, programs to orient institutions and IRBs to CER could help accelerate support for CER. Furthermore, in multicenter studies, the need for IRB approval at multiple sites may sometimes create long delays. One solution is to arrange for IRBs to be able to cede to one another, an approach used in some projects led by members of the Health Maintenance Organization Research Network.

Summary and Recommendations

This NHLBI Expert Workshop highlighted areas that need strengthening to optimize the potential of CER in pulmonary diseases, sleep, and critical care. Key recommendations for agencies and investigators are:

1. Promote interest in CER among the scientific community, NIH research programs, and existing disease research networks or interest groups.

2. Increase efforts to engage stakeholders, including patients, providers, and payers, as participants in the development of CER questions and study designs.

3. Invest in the further development of databases and other infrastructure to enable CER. This includes establishing efficient methods for data sharing while still protecting patient privacy.

4. Make full use of a broad range of study designs, selecting the design that best addresses the need for timeliness and rigor. Designs include but are not limited to cluster randomized trials, pragmatic clinical trials, quasi-experimental studies, and other observational designs.

5. Increase the appropriate use of observational studies, both to conduct CER and to identify areas where CER is needed.

6. Encourage investigators to collect information on a range of outcomes that are important to varying stakeholders, including clinical outcomes, patient-reported outcomes, and economic outcomes.

7. Ensure that committees that review CER grant applications consist of reviewers who have appropriate perspective and training. This includes methodologic expertise in the broad range of study designs described above.

8. Further develop the workforce for CER by supporting training opportunities that focus on the appropriate methodologic and practical skills.

The workshop participants generated a lengthy list of potential CER questions for lung diseases and sleep disorders, but agreed that it was premature to prioritize among them. Rather, Table 3 presents illustrative examples of CER research topics, which are not intended to be all-inclusive. CER has a unique and potentially powerful role in the spectrum of research to enhance health. It is early in its evolution as a research field. Focused efforts to enhance the quality and use of CER in lung diseases are likely to yield rich downstream benefits by improving clinical practice and health care systems and ultimately improving patient health.

TABLE 3.

ILLUSTRATIVE EXAMPLES OF COMPARATIVE EFFECTIVENESS RESEARCH TOPICS FOR LUNG DISEASES AND SLEEP DISORDERS

Lung Condition	Sample CER Research Topic	Some Design Considerations
Asthma	Compare alternative treatments (inhaled corticosteroids, leukotriene receptor antagonists, and combination ICS and β-agonists) in children (12–60 months old) who experience frequent severe wheezing episodes.	Stratify study population by asthma phenotypes and biomarkers, including glucocorticoid responsiveness, to identify predictors of variations in response to therapy.
	Compare adjunctive treatments (behavioral intervention using electronic message prompts to enhance adherence vs. LABA) in patients 12–20 years old who have asthma poorly controlled on low to moderate dose of inhaled corticosteroids alone.
	Compare alternative strategies to step-down from a LABA after the patient's asthma becomes well controlled.	Gradually reduce dose of LABA while sustaining ICS dose, remove LABA and increase ICS dose, remove LABA and add either tiotropium or leukotriene receptor antagonist.
COPD	Compare usual care to usual care plus an evidence-based checklist for treatment to improve the quality and outcomes of care among patients with COPD exacerbations.
	Compare in patients who had a hospital admission for COPD exacerbation the effect on functional status and exacerbation relapse of usual care vs. discharge to a disease self-management pulmonary rehabilitation program.
	What is the comparative effectiveness of using spirometry and symptoms vs. symptoms alone in improving patient-centric outcomes?
Cystic fibrosis	Compare effectiveness of new inhaled antibiotics aztreonam vs. tobramycin as combination therapy vs. cycling or alternating treatment.
	Compare effectiveness of dornase alfa vs. hypertonic saline for chronic treatment of children more than 6 years old.
Acute respiratory care	Compare effectiveness of comprehensive interventions during and after critical respiratory illness treated in the Intensive Care Unit to usual care in improving health outcomes, functional activity levels, and quality of life.	Comprehensive interventions would include physical therapy and pharmacological interventions during critical illness (e.g. neuromuscular electrical stimulation, glucocorticoids, somatostatin to reduce muscular atrophy) and follow up care by multidisciplinary teams (e.g. psychological, physical and occupational therapy specialists)
Sleep apnea	Compare effectiveness, using quasi-experimental methodology (pre vs. post), of a recent CMS policy change that requires providers to track adherence to continuous positive airway pressure through electronic health records on health outcomes, quality of life, and health care use.

Definition of abbreviations: CER = comparative effectiveness research; COPD = chronic obstructive pulmonary disease; ICS = inhaled corticosteroids; LABA = long-acting β-agonists; CMS = Centers for Medicare and Medicaid Services.