Abstract
Comparative effectiveness research (CER) is intended to address the expressed needs of patients, clinicians, and other stakeholders. Representatives of 54 stakeholder groups with an interest in chronic obstructive pulmonary disease (COPD) participated in workshops convened by the COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) over a 2-year period. Year 1 focused on chronic care and care coordination. Year 2 focused on acute care and transitions in care between healthcare settings. Discussions and provisional voting were conducted via teleconferences and e-mail exchanges before the workshop. Final prioritization votes occurred after in-person discussions at the workshop. We used a modified Delphi approach to facilitate discussions and consensus building. To more easily quantify preferences and to evaluate the internal consistency of rankings, the Analytic Hierarchy Process was incorporated in Year 2. Results of preworkshop and final workshop voting often differed, suggesting that prioritization efforts relying solely on requests for topics from stakeholder groups without in-person discussion may provide different research priorities. Research priorities varied across stakeholder groups, but generally focused on studies to evaluate different approaches to healthcare delivery (e.g., spirometry for diagnosis and treatment, integrated healthcare strategies during transitions in care) rather than head-to-head comparisons of medications. This research agenda may help to inform groups intending to respond to CER funding opportunities in COPD. The methodologies used, detailed in the online supplement, may also help to inform prioritization efforts for CER in other health conditions.
Keywords: health services research, research priorities, care coordination, stakeholders
Limited information exists regarding the comparative harms and benefits of treatment alternatives to guide healthcare decision making by patients, clinicians, and other stakeholders in usual practice settings (1). Also, healthcare costs have continued to increase as new and often more expensive treatments are adopted into practice, often without adequate evidence to support their use over existing alternatives (2). These considerations have sparked interest in comparative effectiveness research (CER), which consists of head-to-head comparisons of interventions, including pharmacologic agents, devices, and strategies for delivering healthcare (3). Several federally-funded programs have accelerated CER in the United States, including the Effective Health Care Program (2003), the American Recovery and Reinvestment Act (2009), and, most recently, provisions in the Affordable Care Act (2010), which authorized the creation of the Patient-Centered Outcomes Research Institute (4, 5). Funding opportunity announcements for CER, including those by Patient-Centered Outcomes Research Institute, have emphasized the importance of conducting studies that directly address the needs of patients, clinicians, and other stakeholders who use health-related information to ensure that the most important gaps in evidence for decision making are addressed. The Division of Lung Diseases of the National Heart, Lung, and Blood Institute convened a workshop on CER and also recommended engaging a diverse group of stakeholders to identify and prioritize questions for CER (6).
Chronic obstructive pulmonary disease (COPD) is a lung condition characterized by airflow limitation, progressive dyspnea, and cough, and is punctuated by episodic deteriorations (exacerbations). COPD is a leading cause of morbidity and mortality worldwide (7, 8) and the third leading cause of death in the United States (9, 10). A majority of patients with COPD have at least one other chronic condition (e.g., hypertension, heart failure, depression), receive care from more than one healthcare provider (e.g., primary care and specialist physicians, nurses, respiratory therapists), and cycle between chronic and acute care settings (e.g., outpatient clinic, hospital). Thus, CER topics for COPD need to address different aspects of COPD healthcare, including chronic care, care coordination, acute care, and transitions in care between healthcare settings.
Representatives of diverse stakeholder groups with an interest in COPD participated in premeeting teleconferences and in-person workshops convened by the COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) over a 2-year period (11). Stakeholders included patient advocacy groups, private health plans/payers, physician and nonphysician professional organizations, quality improvement organizations, industry, research organizations, and government agencies that fund research (Table 1). Participants were not asked to disclose potential conflicts of interest (clinical, intellectual, or financial), but their biographies and roles in the stakeholder organization they were representing were included in meeting materials provided to all participants. Stakeholders were asked to develop a CER agenda for COPD in each of the following four areas: (1) chronic care, (2) care coordination, (3) acute care, and (4) transitions in care between healthcare settings. The workshop in Year 1 consisted of research prioritization activities for chronic care and care coordination, whereas the workshop in Year 2 focused on acute care and transitions in care for patients with COPD.
TABLE 1.
STAKEHOLDER GROUPS PARTICIPATING IN CHRONIC OBSTRUCTIVE PULMONARY DISEASE OUTCOMES-BASED NETWORK FOR CLINICAL EFFECTIVENESS AND RESEARCH TRANSLATION WORKSHOPS
| Group | Representatives |
| Patient advocacy groups | |
| Alpha 1 Foundation | Robert Sandhaus, M.D., Ph.D.* |
| Respiratory Health Association of Metropolitan Chicago | Stacy Ignoffo, M.S.W.*† |
| COPD Foundation | Margaret Brown, M.S.†‡ |
| John Walsh*† | |
| Health plans | |
| Blue Cross/Blue Shield | Mike Belman, M.D., M.P.H.* |
| Mark Mattingly, M.D.† | |
| Health Maintenance Organization Research Network | Charlene McEvoy, M.D., M.P.H.*† |
| Douglas Mapel, M.D., M.P.H.*†‡ | |
| Kaiser Permanente | Richard Mularski, M.D., M.S.H.S., M.C.R.*† |
| Wellpoint | John Whitney, M.D.† |
| Professional organizations representing clinicians | |
| American Academy of Family Physicians | Wilson Pace, M.D.† |
| American Academy of Sleep Medicine | Patrick Strollo, Jr., M.D.*† |
| American Association of Cardiovascular and Pulmonary Rehabilitation | Andrew Ries, M.D., M.P.H.*† |
| American Association of Respiratory Care | Sam Giordano, R.R.T., M.B.A.*† |
| American Association of Critical Care Nurses | Mary Lou Sole, Ph.D., R.N.† |
| American College of Physicians | Joe Ramsdell, M.D.* |
| American College of Chest Physicians | Sandra Adams, M.D., M.S.*† |
| American College of Emergency Physicians | Rita Cydulka, M.D., M.S.* |
| American Geriatrics Society | Carlos Fragoso, M.D.† |
| American Public Health Association | Judi Walden, R.C.P. II, R.R.T.* |
| Bunny Newman, R.N.*‡ | |
| American Thoracic Society | David Au, M.D., M.S.*† |
| Lynn Reinke, Ph.D., A.R.N.P.†‡ | |
| Association of Pulmonary and Critical Care Program Directors | Brian Carlin, M.D.† |
| National Association of Social Workers | Joan Levy Zlotnik, Ph.D., A.C.S.W.† |
| Respiratory Nursing Society | Donna Bond, D.N.P., R.N.†‡ |
| Anne Boyle, Ph.D., R.N.† | |
| Diane Locke, A.N.P.-B.C., R.N.†‡ | |
| Society for Academic Emergency Medicine | Eddy Lang, M.D.†‡ |
| Society of General Internal Medicine | Chayan Chakraborti, M.D.† |
| Society of Critical Care Medicine | Jonathan Sevransky, M.D., M.H.S.*† |
| Society of Hospital Medicine | Steven Deitelzweig, M.D.† |
| Greg Maynard, M.D., M.Sc.* | |
| Robert Ryder, M.D.*‡ | |
| Quality improvement/accreditation organizations | |
| American Health Quality Association | Dale Bratzler, D.O., M.P.H.† |
| The Joint Commission | Lee Ann Baggott, M.D.* |
| Tim Byrum, M.S.N., N.P.† | |
| National Patient Safety Foundation | Manisha Shah, M.D.† |
| Industry | |
| Boehringer Ingelheim | Asif Shaikh, M.D., M.P.H.*‡ |
| Researcher organizations | |
| Baystate Health | Peter Lindenauer, M.D., M.S.*† |
| Canadian Association of Emergency Physicians Research Consortium | Brian Rowe, M.D., M.Sc.* |
| North American Primary Care Research Group | Andrew Cave, M.B.*† |
| Ohio State University | Gerene Bauldoff, Ph.D., R.N.* |
| Olmstead Medical Center | Barbara Yawn, M.D., M.Sc.*† |
| Pittsburgh Regional Health Initiative | Keith Kanel, M.D.†‡ |
| Society for Medical Decision Making | Kathryn McDonald, M.M.† |
| The Center for Health Research, Kaiser Permanente | Mary Ann McBurnie, Ph.D.*† |
| William Vollmer, Ph.D.*†‡ | |
| University of Chicago | Jerry A. Krishnan, M.D., Ph.D.*† |
| Edward T. Naureckas, M.D.†‡ | |
| University of Illinois | Todd Lee, Pharm.D., Ph.D.*† |
| University of North Carolina | Shannon S. Carson, M.D.*† |
| Speakers | |
| Johns Hopkins University | Cynthia Boyd, M.D., M.P.H.*‡ |
| Robert Wise, M.D.†‡ | |
| Harvard University | Bartolome Celli, M.D.*‡ |
| National Center for Rural Health Professions | Michael Glasser, Ph.D.*‡ |
| The Joint Commission | Caroline Isbey, R.N., M.S.N., C.D.E.*‡ |
| RTI International | Kenneth LaBresh, M.D.†‡ |
| Denver Health Medical Center | Fred Masoudi, M.D., M.S.P.H.*‡ |
| University of Alberta | Brian Rowe, M.D., M.Sc.†‡ |
| Northwestern University | Mark Williams, M.D.†‡ |
| Government agencies | |
| Agency of Healthcare Research and Quality | Anne Elixhauser, Ph.D.†‡ |
| Supriya Janakiraman, M.D., M.P.H.*†‡ | |
| Centers for Medicaid and Medicare Services | Mark Levine, M.D.*†‡ |
| National Heart, Lung, and Blood Institute | Antonello Punturieri, M.D., Ph.D.†‡ |
| National Institute of Nursing Research | Karen Huss, Ph.D., R.N.†‡ |
| National Institute on Aging | Basil Eldadah, M.D.†‡ |
| External advisory committee | |
| Oregon Health Sciences University | A. Sonia Buist, M.D.*†‡ |
| Johns Hopkins University | Cynthia S. Rand, Ph.D.*†‡ |
| VA Puget Sound Health Care System | Steve Fihn, M.D., M.P.H.*†‡ |
| University of Calgary | Eddy Lang, M.D.*†‡ |
| University of Kentucky | David Mannino, M.D.*†‡ |
| Columbia University | Peter Wyer, M.D.*†‡ |
Representatives of 54 stakeholder groups participated in Year 1, Year 2, or both; in some cases, there was more than one representative from a stakeholder group. Twelve groups were invited but declined or did not respond to the invitation in time to participate: American Academy of Emergency Medicine, American Heart Association, American Lung Association, American Medical Association, American Psychiatric Association, Disease Management Association of America, Improvement Science Research Network, Institute for Health Care Improvement, National Gerontological Nursing Association, PhRMA, United Health Care, and U.S. Food and Drug Administration. The members of the External Advisory Committee are also listed.
Participated in Year 1.
Participated in Year 2.
Nonvoting participant (only one voting representative of each stakeholder group was permitted; speakers, representatives of industry, members of the External Advisory Committee, and government agencies did not vote on priorities but were invited to contribute to discussions).
In this workshop report, we discuss the methodology and development of a national CER agenda for COPD, which may help to inform groups intending to respond to funding opportunities for CER in COPD. A detailed description of the methodology, including topic selection, voting procedures, and discussion about the strengths and limitations of the methodology, are available in the online supplement. In brief, stakeholders participated in preworkshop discussions to plan the meeting and to develop a provisional list of research priorities. Stakeholders then voted to identify provisional CER priorities before the in-person workshop, and the provisional topics were included in meeting material provided to participants (see Tables E2 and E3 in the online supplement). At the Year 1 and 2 workshops, we used the modified Delphi approach (12, 13), a standard approach to consensus development. At the conclusion of the Year 1 workshop, attendees completed a 10-question online satisfaction survey to provide anonymous feedback regarding the format and content of the meeting and to inform the planning for the second workshop. At the Year 2 workshop, we used the analytic hierarchy process (AHP; 14–16) to more easily quantify relative preferences for the various CER topics and to assess the internal consistency of voting. At the conclusion of the Year 2 workshop, stakeholders were asked to vote on the importance of various criteria used to set CER priorities. Before finalizing this workshop report, stakeholders and members of an External Advisory Committee (see Acknowledgments) were invited to submit comments to a draft containing the prioritized research agenda.
Stakeholder Participation
Fifty-four stakeholder groups participated in the workshops (82% of 66 invited groups; Table 1), including 3 patient advocacy groups (4 invited), 4 health plans (5 invited), 18 professional organizations representing clinicians (24 invited), 3 quality improvement/accreditation organizations (4 invited), 1 representative of industry (2 invited), 11 research organizations (12 invited), 9 speakers (9 invited), and 5 government agencies (6 invited). Of the total 54 stakeholder groups invited for Year 1 and 2 workshops, 33 groups participated in Year 1 (38 individuals) and 42 groups participated in Year 2 (51 individuals). We did not collect information about why stakeholder groups participated in one but not both workshops. In cases in which there was more than one representative from a single stakeholder group, only one voting representative was permitted. The draft research questions and rationale proposed by stakeholders for each CER topic are provided in Tables E1 to E3. We present below the priorities for CER topics in each of the four areas (chronic care, care coordination, acute care, and transitions in care).
Priorities for Comparative Effectiveness Research in Copd
Chronic Care
Nine topics for chronic care were identified (Table 2). Preworkshop voting results indicated that all topics were important, with little separation (median importance scores 2–4; possible range 1–9). After discussions at the workshop, stakeholders were asked to rank the importance of topics to more easily distinguish between preferences. Final voting results indicated that Spirometry for diagnosis and treatment was the highest-priority topic (median rank 1). However, there remained substantial overlap in priorities. For example, three topics (Effectiveness of pulmonary rehabilitation, Effectiveness of COPD care and guideline care translation, and Home care) all had a median rank of 4, the second-highest priority rank. The interquartile ranges (IQRs) for ranks were also relatively wide, suggesting variable preferences among stakeholder groups.
TABLE 2.
RESULTS OF YEAR 1 WORKSHOP: CHRONIC CARE FOR CHRONIC OBSTRUCTIVE PULMONARY DISEASE TOPICS
| Preworkshop Voting | Final Workshop Voting | ||
| Importance Scores | Importance Ranks | ||
| 1 (Very Important) to 9 (Most Unimportant) | 1 (Most Important) to 9 (Most Unimportant) | ||
| Topic | Chronic COPD Care | Median Score (IQR) | Median Rank (IQR) |
| 1 | Spirometry for diagnosis and treatment | 3 (2–5) | 1 (1–4) |
| 2 | Effectiveness of pulmonary rehabilitation | 3 (1–3) | 4 (2–5) |
| 3 | Effectiveness of COPD care and guideline translation | 2 (1–3) | 4 (1–6) |
| 4 | Home care | 4 (3–5) | 4 (3–6) |
| 5 | Development of performance measures | 3 (1–4) | 5 (3–7) |
| 6 | Oxygen therapy | 2 (1–3) | 5 (3–8) |
| 7 | Palliation of symptoms | 3 (2–4) | 7 (5–7) |
| 8 | Pharmaceutical treatment | 3 (3–4) | 7 (6–8) |
| 9 | Smoking cessation | 2 (1–3) | 8 (6–8) |
Definition of abbreviations: CER = comparative effectiveness research; COPD = chronic obstructive pulmonary disease; IQR = interquartile range.
There were 33 participating stakeholder organizations in the workshop; individual participants are listed in Table 1. Preferences in the Preworkshop Voting were expressed as importance scores, whereas importance ranks were used for the Final Workshop voting. Ranks were used in an attempt to more easily separate topics by level of importance. To more clearly observe differences in priorities within each CER area, stakeholders were informed that no two topics should be assigned the same importance rank. Wide IQRs suggest variation in stakeholder preferences.
Care Coordination
Eleven care coordination topics were identified (Table 3). As with chronic care topics, preworkshop voting results rated all care coordination topics as important. Final voting, using ranks, identified Management of COPD in the presence of comorbidity and Pulmonary rehabilitation as a model for care coordination as the highest two priorities (median ranks 3 and 3.5, respectively). However, none of the care coordination topics achieved a median rank of 1, and the IQRs for ranks were wide, indicating again substantial variability in preferences across stakeholder groups.
TABLE 3.
RESULTS OF YEAR 1 WORKSHOP: CARE COORDINATION TOPICS
| Preworkshop Voting | Final Workshop Voting | ||
| Importance Scores | Importance Ranks | ||
| 1 (Very Important) to 9 (Most Unimportant) | 1 (Most Important) to 11 (Most Unimportant) | ||
| Topic | Care Coordination | Median Score (IQR) | Median Rank (IQR) |
| 1 | Management of COPD in the presence of comorbidity | 2 (1–3) | 3 (2–5) |
| 2 | Pulmonary rehabilitation as model for care coordination | 3 (1–3) | 3.5 (2–8) |
| 3 | Impact of depression and mental health management | 3 (1–3) | 5 (4–8) |
| 4 | Measurement of quality of care coordination | 3 (2–4) | 6 (2–7) |
| 5 | Quality of care coordination | 3 (2–3) | 6 (3–8) |
| 6 | Comprehensive COPD patient education | 2 (1–3) | 6.5 (4–9) |
| 7 | Cost effectiveness of care coordination | 3 (2–4) | 6.5 (5–10) |
| 8 | Case management | 3 (2–5) | 7 (5–8) |
| 9 | Developing a tool for risk satisfaction of patient COPD | 3 (3–4) | 7 (3–9) |
| 10 | Patient-centered medical home | 3 (2–4) | 7.5 (4–10) |
| 11 | Scheduled multidisciplinary team consultation | 3 (2–3) | 9 (8–10) |
Definition of abbreviations: CER = comparative effectiveness research; COPD = chronic obstructive pulmonary disease; IQR = interquartile range.
There were 33 participating stakeholder organizations in the workshop; individual participants are listed in Table 1. Preferences in the Preworkshop Voting were expressed as importance scores, whereas importance ranks were used for the Final Workshop voting. Ranks were used in an attempt to more easily separate topics by level of importance. To more clearly observe differences in priorities within each CER area, stakeholders were informed that no two topics should be assigned the same importance rank. Wide IQRs suggest variation in stakeholder preferences.
Acute Care
Eight topics for acute care (Table 4) were identified. Despite use of ranks for preworkshop voting, several topics had similar levels of importance (e.g., five topics had median ranks of 3–4 for acute COPD care).
TABLE 4.
RESULTS OF YEAR 2 WORKSHOP: ACUTE CHRONIC OBSTRUCTIVE PULMONARY DISEASE CARE
| Preworkshop Voting |
Final Workshop Voting |
||||
| Importance Ranks |
Normalized and Idealized Priorities |
||||
| 1 (Most Important) to 8 (Most Unimportant) |
Priority Scores Using AHP |
||||
| Topic | Acute COPD Care | Median Rank (IQR) | Median Normalized Priority | Normalized Priority IQR | Idealized Priority |
| 1 | Implementation of COPD exacerbation checklist | 4 (2–6) | 0.22 | (0.12–0.26) | 1.00 (reference) |
| 2 | Comorbid conditions in the acute setting | 3 (2–5) | 0.20 | (0.06–0.36) | 0.95 |
| 3 | Effectiveness and implementation of NIV in acute respiratory failure | 4 (2–5) | 0.17 | (0.07–0.25) | 0.80 |
| 4 | Role of antibiotics in acute exacerbations | 4 (3–6) | 0.14 | (0.06–0.14) | 0.63 |
| 5 | Quality of care assessment | 4 (2–6) | 0.12 | (0.04–0.19) | 0.57 |
| 6 | Dyspnea in COPD | 5 (3–6) | 0.07 | (0.03–0.08) | 0.31 |
| 7 | Oxygen therapy post exacerbation | 7 (4–8) | 0.04 | (0.02–0.04) | 0.19 |
| 8 | Interventions for acute airway obstruction | 6 (4–7) | 0.04 | (0.02–0.04) | 0.18 |
Definition of abbreviations: AHP = analytic hierarchy process; COPD = chronic obstructive pulmonary disease; IQR = interquartile range; NIV = noninvasive ventilation.
There were 42 participating stakeholder organizations in the workshop; individual participants are listed in Table 1. Wide IQRs suggest variation in stakeholder preferences. Preferences in the preworkshop voting were expressed as importance ranks, whereas AHP was used to develop normalized and idealized priorities in the final voting. AHP was used in an attempt to quantify the level of separation in importance across ranked topics. The normalized priorities represent the proportion of the total importance across all topics that voters ascribe to a particular research topic. The ratio of normalized priorities for any two topics indicates their relative importance (idealized priority). Thus, the AHP approach ranks topics from most important to least important, but the process also offers the advantage of quantifying the separation between any two options (e.g., how much more important is the most highly ranked topic compared to any other topic).
Final voting results, using AHP to rank topics, indicated that Implementation of COPD exacerbation checklist, Comorbid conditions in the acute setting, and Effectiveness and implementation of noninvasive ventilation (NIV) were the most highly voted topics. These three topics had very similar idealized priority scores (1.0 [reference], 0.95, 0.80, respectively), indicating that Comorbid conditions and NIV topics were voted as at least 80% as important as Implementation of COPD exacerbation checklist. By contrast, the two lowest ranked topics (Oxygen therapy postexacerbation and Interventions for acute airway obstruction) were viewed as one-fifth as important (idealized priority scores about 0.2).
Transitions in Care
Seven topics for transitions in care were identified (Table 5). Final voting results, again using AHP, indicated that the highest priority topic, Integrated healthcare strategies during transitions in COPD care (e.g., early hospital discharge to pulmonary rehabilitation) was rated as more than twice as important as the second-highest preferences (Multimorbidity and Patient and family activation).
TABLE 5.
RESULTS OF YEAR 2 WORKSHOP: TRANSITIONS IN CHRONIC OBSTRUCTIVE PULMONARY DISEASE CARE
| Preworkshop Voting |
Final Workshop Voting |
||||
| Importance Ranks |
Normalized and Idealized Priorities |
||||
| 1 (Most Important) to 9 (Most Unimportant) |
Priority Scores Using AHP |
||||
| Topic | Transitions in COPD Care | Median Rank (IQR) | Median Normalized Priority | Normalized Priority IQR | Idealized Priority |
| 1 | Integrated healthcare strategies during transitions in COPD care | 2 (1–3) | 0.40 | (0.23–0.52) | 1.00 (reference) |
| 2 | Multimorbidity and transitions in COPD care | 3 (2–4) | 0.16 | (0.07–0.24) | 0.40 |
| 3 | Patient and family activation during transitions in COPD care | 4 (3–5) | 0.15 | (0.06–0.20) | 0.38 |
| 4 | End of life/palliative care and transitions in COPD care | 4 (2–6) | 0.12 | (0.05–0.20) | 0.31 |
| 5 | Early diagnosis and treatment of COPD | 5 (2–6) | N/A* | N/A* | N/A* |
| 6 | Psychosocial barriers during transitions in COPD care | 5 (4–6) | 0.09 | (0.04–0.12) | 0.22 |
| 7 | Modeling effects of interventions and transitions in COPD status | 6 (4–7) | 0.08 | (0.03–0.11) | 0.22 |
Definition of abbreviations: AHP = analytic hierarchy process; COPD = chronic obstructive pulmonary disease; IQR = interquartile range; N/A = not applicable.
There were 42 participating stakeholder organizations in the workshop; individual participants are listed in Table 1. Wide IQRs suggest variation in stakeholder preferences. Preferences in the preworkshop voting were expressed as importance ranks, whereas AHP was used to develop normalized and idealized priorities in the final voting. AHP was used in an attempt to quantify the level of separation in importance across ranked topics. The normalized priorities represent the proportion of the total importance across all topics that voters ascribe to a particular research topic. The ratio of normalized priorities for any two topics indicates their relative importance (idealized priority). Thus, the AHP approach ranks topics from most important to least important, but the process also offers the advantage of quantifying the separation between any two options (e.g., how much more important is the most highly ranked topic compared to any other topic).
Topic 5 (Early diagnosis and treatment of COPD) for Transitions in COPD care was not voted on at the workshop because it was folded into Topic 3 (Patient and family activation during transitions in COPD care) after discussion at the in-person meeting.
The consistency ratio (CR) for voting patterns for acute care topics and for transition in care topics was less than 0.1 for each, suggesting that expressed preferences by stakeholders were overall internally consistent. See online supplement for further explanation.
Satisfaction With Workshops
Feedback indicated high levels of satisfaction among workshop participants (Table 6). Overall, Year 1 workshop attendees supported the need for a 2-day meeting, with presentations on the first day framing the subsequent day’s discussion and voting. A similar format was therefore used in Year 2.
TABLE 6.
SATISFACTION SURVEY COMPLETED BY STAKEHOLDERS
| 1 (Strongly Disagree) to 5 (Strongly Agree) | |
| 1. The workshop met my expectations. | 4 (4–5) |
| 2. I feel that the products of the workshop will have impact. | 4 (4–5) |
| 3. The format and content of the first day was very useful. | 4 (4–5) |
| 4. I would recommend less lecture time for Day 1. | 3 (2–4) |
| 5. I would recommend more discussion time for Day 1. | 4 (3–4) |
| 6. The format and content of the second day was very useful. | 4 (3–4) |
| 7. I believe that the workshop objectives could be met in a 1-day instead of a 2-day workshop. | 2 (2–2.3) |
| 8. I would recommend others to participate in this event. | 4 (3–4) |
| 9. If given the opportunity, I would participate in this event again. | 5 (4–5) |
| 10. Overall, I was very satisfied with this workshop. | 4 (4–5) |
Definition of abbreviations: CONCERT = COPD Outcomes-based Network for Clinical Effectiveness and Research Translation; IQR = interquartile range.
Data are presented as median (IQR). Stakeholders were asked to complete an anonymous satisfaction survey at the end of the Year 1 workshop to help CONCERT plan for the workshop in Year 2. Thirty-three stakeholder organizations completed the satisfaction survey.
Criteria Used by Stakeholders for Prioritizing Cer Topics
Stakeholders were asked to rank the importance of seven criteria when making decisions about priorities for CER (Table 7). Twenty-three (55% of 42 groups participating in Year 2) completed ballots describing the criteria used to set CER priorities, so results should be viewed cautiously. The Impact on patient-centered health outcomes was voted as most important, followed by Quality of the evidence, Societal costs of care, and Variability in care.
TABLE 7.
CRITERIA USED BY STAKEHOLDERS TO SELECT COMPARATIVE EFFECTIVENESS RESEARCH TOPICS
| Criterion | 1 (Most Important) to 7 (Most Unimportant) Importance Rank |
| 1. Impact on patient-centered health outcomes in efficacy studies (magnitude of benefit demonstrated in efficacy studies and likelihood to guide daily practice) | 1 (1–2.5) |
| 2. Quality of the evidence about patient-centered outcomes in efficacy studies | 3 (2–4.5) |
| 3. Societal costs of care | 4 (3–5) |
| 4. Variability in care, including health disparities in care | 4 (3–5.5) |
| 5. Potential for a difference between efficacy and effectiveness | 5 (2.5–6) |
| 6. Feasibility of effectiveness studies and implementation studies | 5 (4–6) |
| 7. Informs healthcare across diverse areas | 5 (4–7) |
Data are presented as median rank (interquartile range). Twenty-three stakeholder organizations completed a survey on criteria used to select comparative effectiveness research topics after the final prioritization vote at the Year 2 workshop.
Lessons Learned and Recommendations for Cer in Copd
Four primary findings emerged from the workshops to elicit national priorities for CER in COPD: (1) engaging diverse stakeholders is feasible and can be used to identify support for CER topics; (2) efforts to develop and prioritize a research agenda based solely on requests for topics from stakeholder groups without discussion may provide different priorities for CER compared with efforts that incorporate discussion; (3) preferences for CER and criteria used to select CER topics varied across stakeholder groups; however, stakeholder support was strongest for studies to evaluate and to improve healthcare delivery in usual practice settings; and (4) the AHP helped to quantify the relative preferences for CER and to evaluate the internal consistency of stakeholder voting patterns.
A diverse group of 54 stakeholders representing patients, clinicians, and other healthcare decision makers participated in the development of the national research agenda for CER in COPD. Survey results indicated high levels of support for the approach used by CONCERT, including preworkshop discussions to plan the meeting and elicit a provisional list of research priorities, followed by in-person workshops with presentations and discussion before final voting. We suspect that stakeholders benefited from a structured approach for information sharing and to work with those with differing perspectives. A limitation of the methodology used in setting research priorities is the potential for conflicts of interests (e.g., clinical, intellectual, or financial) to influence voting patterns of individual participants. We were not aware of standards regarding the disclosure and management of potential conflicts of interests when setting research priorities when we conducted the workshops. There is emerging consensus on the need for disclosure and management of potential conflicts of interest when developing clinical practice guidelines; a similar need exists for groups developing research priorities.
Other groups, including the Institute of Medicine Committee on Comparative Effectiveness Research Prioritization and the AHRQ Community Forum, have engaged stakeholders in developing CER priorities (but none specifically for COPD) based on questionnaires, written comments, and oral presentations (1, 17). By contrast, the methodology used by CONCERT included sustained interactions over a 2-year period. We used a modified Delphi approach to facilitate discussions and consensus building during the in-person workshops; it is therefore not surprising that results of preworkshop votes and final votes at the workshops differed. These findings suggest that prioritization efforts relying solely on requests for topics from stakeholder groups without discussion may provide different priorities for CER compared with efforts that incorporate discussion. The sustained interest among COPD stakeholders to participate in prioritization efforts over this 2-year period also raises the possibility for additional multistakeholder collaborations, including the development of specific research proposals (e.g., translating research topics into specific research questions and study designs), dissemination of results, and implementation of findings.
In a previous effort to identify CER priorities for COPD over a shorter (2 mo) period, we asked clinicians and researchers to nominate potential research topics and then convened a separate meeting with a limited number of other stakeholder groups (two health plans, one patient advocacy group, two government agencies, and eight professional societies) for a final overall prioritization vote (18). We observed substantial variation in stakeholder preferences, with none achieving an overall rank of 1 (most important CER topic). In the current effort, we sought to develop consensus by promoting and sustaining interactions across all stakeholder groups and by developing a separate list of CER priorities for different aspects of COPD care (i.e., chronic care, care coordination, acute care, transitions in care). We also involved a larger number and more diverse group of stakeholders (e.g., organizations involved in quality improvement or accreditation) and promoted dialogue with experts in effectiveness and implementation research in the current effort.
Studies to evaluate and to improve healthcare delivery received the greatest levels of support, rather than head-to-head comparisons of specific medications. However, preferences for CER varied across stakeholder groups even within a specific aspect of COPD care (e.g., chronic care), likely because decision making appears to be governed by a complex matrix of criteria. Stakeholders indicated that the Impact on patient-centered health outcomes was the most important criterion, although a number of other criteria were also used when selecting among different COPD CER topics. Importantly, the relative importance of these criteria in setting research priorities varied across stakeholder groups. Collectively, these observations suggest that opportunities to reach uniform consensus for a single research priority in CER are limited; it is, however, feasible to differentiate the higher (vs. lower) priority topics and to build a pipeline of stakeholder-supported CER. Our observations also suggest that stakeholder support for CER topics may vary depending on which groups are invited to participate in setting priorities. This issue highlights the importance of engaging a diverse group of stakeholders to ensure broad input, which may need to be enriched further depending on the purpose (e.g., greater involvement of patients for applications in response to funding opportunities in response to the Patient Centered Outcomes Research Institute). The workshops included only one representative of industry, despite our efforts to attract greater involvement by contacting PhRMA, the leading trade organization for the pharmaceutical industry and biotechnology companies. Also, other industry representatives (e.g., device manufacturers) were not participants, and future efforts may benefit from including a wider cross-section of industry representatives. The optimal size and breadth of stakeholder organizations for research prioritization activities needs further study (19).
We also found that asking stakeholders to rank rather than simply rate the importance of topics to be more informative. We were able to better quantify the relative preferences for different CER topics using the AHP methodology in Year 2 to calculate “idealized priorities.” The AHP methodology also provided the opportunity to detect inconsistent voting patterns; results were fortunately reassuring. Of note, we used a limited version of the AHP in Year 2 and only asked stakeholders to compare the overall importance of topics. A fully deployed AHP would require: (1) defining and prioritizing the universe of criteria used by stakeholders, (2) using AHP to determine how well each topic satisfies each criterion, and (3) pooling the votes to develop a prioritized list of research topics. For each criterion, there would be n(n − 1)/2 pairwise comparisons, where n is the number of research topics being compared. For the full deployment of the AHP in Year 2, there would then be 196 pairwise comparisons of acute care topics (28 pairwise comparisons of topics per criterion × 7 criteria) and another 147 pairwise comparisons of transitions in care topics, or a total 343 pairwise comparisons. Such an effort was believed to be impractical for a single meeting and suggests the need for defining and selecting a more parsimonious list of criteria and CER topics. Furthermore, only about half the stakeholders provided information about criteria used by their organization for setting research priorities. Findings in this report may help inform future efforts to more fully deploy AHP and thereby more explicitly link the criteria proposed by stakeholders to the selection of CER priorities. It has been 2 years since the last workshop, and priorities in COPD CER may have shifted since then. However, input in CONCERT workshops was broad, and further input can occur through subsequent efforts to update priorities.
Summary and Conclusions of Workshops
We used a standard approach to consensus development, the modified Delphi approach, as well as a novel application of AHP for developing a national CER agenda for COPD in collaboration with a diverse set of stakeholders. Studies to improve healthcare delivery, rather than head-to-head comparisons of medications, were rated as highest priority. As there was substantial variation in preferences among stakeholders for specific research topics, researchers may need to collaborate with different groups of stakeholders depending on which research topics are proposed for study. The various “lessons learned” from the workshops may help to inform similar efforts to identify and prioritize CER topics for other health conditions.
A complete list of stakeholder-supported CER priorities is available in Tables 2 through 5. The rationale and draft research questions linked to the CER topics is available in Tables E1 to E3. Stakeholder preferences were strongest for the following CER topics:
Chronic care: Spirometry for diagnosis and treatment, Effectiveness of pulmonary rehabilitation, Effectiveness of COPD care and guideline translation, and Home care.
Care coordination: Management of COPD in the presence of comorbidity, and Pulmonary rehabilitation as a model for care coordination.
Acute care: Implementation of COPD exacerbation checklist, Comorbid conditions in the acute setting, and Effectiveness and implementation of NIV in acute respiratory failure.
Transitions in care: Integrated healthcare strategies, Multimorbidity, and Patient and family activation.
Supplementary Material
Acknowledgments
The authors thank the members of the Workshop Organizing Committee (Gerene Bauldoff, Ph.D., R.N., Ohio State University; Kenneth W. Lin, M.D., Agency for Healthcare Research and Quality; Barbara Yawn, M.D., M.Sc., Olmsted Medical Center; and CONCERT investigators) and the External Advisory Committee (A. Sonia Buist, M.D., Oregon Health and Sciences University; Stephan Fihn, M.D., M.P.H., Department of Veterans Affairs; Eddy Lang, M.D., University of Calgary; David Mannino, M.D., University of Kentucky School of Medicine; Cynthia S. Rand, Ph.D., Johns Hopkins University; Peter Wyer, M.D., Columbia University) for their help to plan and conduct the workshops and in reviewing earlier drafts of this report. CONCERT also thanks representatives of stakeholder organizations that contributed to the workshops. The authors thank Stephanie Lewis, B.A., Jeffery Charbeneau, M.S., and William T. Summerfelt, Ph.D., for their help in organizing and conducting the meetings, and Jill Pope and Helene Gussin, Ph.D., for their help in editing and finalizing this report.
Footnotes
Supported by Agency for Healthcare Research and Quality grant AHRQ R13 HS017894 and National Institutes of Health/National Heart, Lung, and Blood Institute grant NIH/NHLBI RC2 HL1011618. D.H.A. is supported through Health Services Research and Development, Department of Veterans Affairs.
Author Contributions: J.A.K., P.K.L., D.H.A., S.S.C., T.A.L., M.A.M., E.T.N., W.M.V., and R.A.M. all had substantial contributions as follows: (1) conception and design, acquisition of data, or analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content; and (3) final approval of the version to be published.
The views expressed in this article represent those of the authors and do not necessarily represent those of the Department of Veterans Affairs.
This article has an online supplement, which is accessible from this issue’s table of contents at www.atsjournals.org
Originally Published in Press as DOI: 10.1164/rccm.201206-0994WS on November 15, 2012
Author disclosures are available with the text of this article at www.atsjournals.org.
References
- 1.Institute of Medicine Initial National Priorities for Comparative Effectiveness Research. Washington DC: National Academies Press; June, 2009 [Google Scholar]
- 2.Pearson SD, Bach PB. How Medicare could use comparative effectiveness research in deciding on new coverage and reimbursement. Health Aff (Millwood) 2010;10:1796–1804 [DOI] [PubMed] [Google Scholar]
- 3.U.S. Department of Health and Human Services Report to the president and the congress on comparative effectiveness research. June 30, 2009 [accessed 2012 Jan 8]. Available from: http://www.hhs.gov/recovery/programs/cer/execsummary.html
- 4.One hundred eleventh Congress of the United States of America. The Patient Protection and Affordable Care Act. H.R. 3590. 2010 [accessed 2012 Jan 2]. Available from: http://burgess.house.gov/UploadedFiles/hr3590_health_care_law_2010.pdf.
- 5.Clancy C, Collins FS. Patient-Centered Outcomes Research Institute: the intersection of science and healthcare. Sci Transl Med 2010;2:37cm18 [DOI] [PubMed] [Google Scholar]
- 6.Lieu TA, Au D, Krishnan JA, Moss M, Selker H, Harabin A, Taggart V, Connors A. Comparative effectiveness research in lung disease and sleep disorders: recommendations from the National Heart, Lung, and Blood Institute workshop. Am J Respir Crit Care Med 2011;184:848–856 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mannino DM, Buist AS. Global burden of COPD: risk factors, prevalence, and future trends. Lancet 2007;370:765–773 [DOI] [PubMed] [Google Scholar]
- 8.Global Initiative for Chronic Obstructive Lung Disease. Global Strategy for the Diagnosis, Management and Prevention of COPD (2008 edition) [accessed 2011 Dec 21]. Available from: http://www.goldcopd.org/
- 9.Akinbamani LJ, Liu X. Chronic Obstructive Pulmonary Disease among adults aged 18 and over in the United States, 1998–2009. NCHS data brief, no 63. Hyattsville, MD: National Center for Health Statistics. 2011 [accessed 2012 Jan 2]. Available from: http://www.cdc.gov/nchs/data/databriefs/db63.htm [PubMed]
- 10.U.S. Department of Health and Human Services, National Institutes of Health, National Heart, Lung, and Blood Institute. Morbidity and mortality: 2012 chartbook on cardiovascular, lung and blood diseases [accessed 2013 January 10]. Available from: http://www.nhlbi.nih.gov/resources/docs/2009_ChartBook.pdf [DOI] [PubMed]
- 11.Mularski RA, McBurnie MA, Lindenauer PK, Lee TA, Vollmer WM, Au DH, Carson SS, Krishnan JA; CONCERT investigator consortium Comparative effectiveness research in chronic obstructive pulmonary disease. J Comp Effect Res 2012;1:71–82 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, Marteau T. Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998;2:i–iv, 1–88 [PubMed] [Google Scholar]
- 13.American College of Chest Physicians. ACCP Consensus Statement Development Guide [accessed 2011 Dec 21]. Available from: http://www.accpstorage.org/newOrganization/guidelines/ConsensusGuide.pdf.
- 14.Vaidya OS, Kumar S. Analytic hierarchy process: an overview of applications. Eur J Oper Res 2006;169:1–29 [Google Scholar]
- 15.Saaty TL. The analytic hierarchy process. Revised edition Pittsburgh, PA: RWS Publications; 2000 [Google Scholar]
- 16.Saaty TL. A scaling method for priorities in hierarchical structures. J Math Psychol 1977;15:234–281 [Google Scholar]
- 17.Agency for Healthcare Research and Quality. AHRQ Community Forum [accessed 2012 Sep 3]. Available from: http://effectivehealthcare.ahrq.gov/index.cfm/who-is-involved-in-the-effective-health-care-program1/ahrq-community-forum/
- 18.Pickard AS, Lee TA, Solem CT, Joo MJ, Schumock GT, Krishnan JA. Prioritizing comparative effectiveness research topics via stakeholder involvement: an application in COPD. Clin Pharmacol Ther 2011;90:888–892 [DOI] [PubMed] [Google Scholar]
- 19.Dubois RW, Graff JS. Setting priorities for comparative effectiveness research: from assessing public health benefits to being open with the public. Health Aff 2011;30:2235–2242 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.