Community-acquired Pneumonia Guideline Recommendations—Impact of a Consensus-based Process versus Systematic Reviews

Abstract

Background

The American Thoracic Society (ATS)/Infectious Diseases Society of America (IDSA) Community-acquired Pneumonia (CAP) guidelines were developed using systematic reviews to inform every recommendation, as suggested by the Institute of Medicine Standards for Trustworthy Guidelines. Recent studies suggest that an expert consensus-based approach, called the Convergence of Opinion on Recommendations and Evidence (CORE) process, can produce recommendations that are concordant with recommendations informed by systematic reviews.

Purpose

The goal of the study was to evaluate the efficacy of the CORE process had it been used to develop the ATS/IDSA CAP guidelines.

Methods

Experts in CAP who were not on the guideline panel and had no knowledge of the guideline’s systematic reviews or recommendations were recruited to participate in the CORE process, addressing the same questions asked by the guideline panel. Recommendations derived from the CORE process were compared to the guideline recommendations. Concordance of the course of action, strength of recommendation, and quality of evidence were determined.

Results

Using a threshold of 70% of experts selecting the same course of action to make a recommendation, the CORE process yielded a recommendation for 20 of 31 (65%) questions. Among the 20 CORE-derived recommendations, 19 (95%) were concordant with the guideline recommendations (kappa agreement 0.88, 95% CI .64–1.00). There was less agreement among the strength of recommendations (58%) and quality of evidence (42%).

Conclusions

If the CORE process had been used, 11 systematic reviews would have been necessary rather than 31, with minimal impact on the recommended courses of action.

The ATS/IDSA Guidelines on Community-acquired Pneumonia could have been developed more efficiently without sacrificing accuracy if a Delphi-like process, Convergence of Opinion on Recommendations and Evidence (CORE), was used to determine which questions require a systematic review.

The American Thoracic Society (ATS)/Infectious Diseases Society of America (IDSA) Diagnosis and Treatment of Adults with Community-acquired Pneumonia (CAP) guidelines were developed using systematic reviews to inform every recommendation [1], as suggested by the Institute of Medicine (IOM) Standards for Trustworthy Guidelines [2]. Frequent criticisms of an IOM-adherent approach are 1) the methodological expertise needed to conduct the systematic reviews is expensive and hard to find and 2) performing a systematic review for every question is time-consuming and burdensome. An unintended consequence is that the number of questions addressed by a guideline must be restricted to finish the guideline in a timely and affordable fashion, limiting the advice that an expert panel can provide to stakeholders.

Two recent studies reported that an expert consensus-based approach, called the Convergence of Opinion on Recommendations and Evidence (CORE) process, yields recommendations that are concordant with recommendations informed by a systematic review, but are developed more efficiently because the systematic review is not required. In one study, the CORE process yielded recommendations for 55 of 67 (80%) questions; 54 of 55 (98%) recommendations were the same as the recommendations from IOM-adherent guidelines [3]. The only discordant recommendation was informed by minimal empirical evidence and remains controversial. In another study, a modified CORE process yielded recommendations for 9 of 10 (90%) guideline questions; 8 of 9 (89%) recommendations were the same as the recommendations from IOM-adherent guidelines, with the lone discordant recommendation occurring for a question for which the guideline panel chose to make “no recommendation” [4]. The studies’ authors concluded that the CORE process may be a reasonable approach to distinguish questions that require a systematic review from questions that do not [3–5]. The CORE process has since been used by international task forces to develop clinical guidance on the management of COVID-19 [6–9]. The goal of this study was to estimate the impact of the CORE process if it had been used to develop the ATS/IDSA CAP guidelines.

METHODS

Participants

CAP Guideline Panel

The co-chairs of the CAP guidelines were selected by the ATS and IDSA, based upon their expertise in the field of CAP and their successful leadership of prior society projects. The co-chairs worked together to select guideline panelists, many of whom had participated in previous guideline projects. All panelists disclosed potential conflicts of interest, which were vetted according to the policies and procedures of the ATS, and then were approved by the sponsoring societies. In addition, there was a methodology team that consisted of an experienced lead methodologist and a methodology trainee. The methodology team performed the systematic reviews that were presented to the guideline panelists for decision-making but did not formulate or vote on recommendations.

CORE Process Panel

Leaders from the ATS’ Assembly on Pulmonary Infections and Tuberculosis identified CAP experts who were not on the guideline panel. Those experts were invited by email to participate (Supplementary Figure 1). The CAP experts confirmed they had no advanced knowledge of the CAP guideline’s systematic reviews, discussions, or recommendations before being approved to participate in the project. They were not asked to disclose potential conflicts of interest.

Guideline Development

The CAP guideline was developed using IOM-adhered methodology [2]. Questions were developed using the Population, Intervention, Comparator, Outcome (PICO) framework and then a systematic search strategy was developed and study selection criteria were determined. The Medline database was searched, studies were selected according to the prespecified selection criteria, and crude data were extracted by the methodologists. Data were pooled by meta-analysis when possible and the quality of the evidence was rated as high, moderate, low, or very low using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach [10]. The methodology team presented the systematic reviews to the expert panel at a face-to-face meeting and several webinars, at which time evidence was discussed and group decision-making occurred. Recommendations were formulated after considering the balance of benefits versus harms, burdens, and cost; quality of evidence; patient values and preferences; and feasibility. The recommendations were rated by the panelists as either strong or conditional (ie, weak) according to the GRADE approach.

CORE Process

The CORE process was conducted after the guideline panel had formulated and graded its recommendations, but before the guideline had been published or presented publicly. One of the guideline co-chairs provided the CAP guideline’s PICO questions to the CORE leader (Table 1). The CORE process has been described in detail elsewhere [3]. Briefly, SurveyMonkey^® software (SurveyMonkey, San Mateo, CA) was used to develop an electronic multiple-choice survey. Each survey question consisted of four parts: 1) presentation of the PICO question, 2) a multiple-choice response asking for a strong or weak recommendation for or against a course of action, or no recommendation, 3) a second multiple-choice question asking participants to evaluate their impression of the quality of the evidence, and 4) a free-text box for comments (Supplementary Figure 2). The survey was administered only once to streamline the process and improve participant satisfaction (the original CORE process included multiple cycles [3]).

Table 1.

PICO Questions for the ATS/IDSA Community-acquired Pneumonia Guidelines

Sputum Gram stain and culture
1	Should outpatient adults with community-acquired pneumonia (CAP) undergo routine sputum Gram stain and culture?
2	Should hospitalized adults with severe CAP undergo routine sputum Gram stain and culture?
Blood cultures
3	Should outpatient adults with CAP undergo routine blood cultures prior to the initiation of antibiotic treatment?
4	Should hospitalized adults with nonsevere CAP and no risk factors for drug-resistant pathogens undergo routine blood cultures prior to the initiation of antibiotic treatment?
5	Should hospitalized adults with severe CAP undergo routine blood cultures prior to the initiation of antibiotic treatment?
Antigen testing
6	Should adults who have nonsevere CAP undergo routine rapid testing of the urine for pneumococcal antigen?
7	Should adults who have severe CAP undergo routine rapid testing of the urine for pneumococcal antigen?
8	Should adults who have nonsevere CAP undergo routine rapid testing of the urine for legionella antigen?
9	Should adults who have severe CAP and/or risk factors for legionella infection undergo routine rapid testing of the urine for legionella antigen?
Other diagnostic testing
10	Should adults who have severe CAP and/or risk factors for legionella infection have their respiratory secretions either cultured on selective media or tested by nucleic acid amplification?
11	Should adults who have CAP and come from a community in which influenza cases have been reported undergo routine testing for influenza (either PCR testing alone or rapid influenza antigen testing followed by PCR testing)?
12	In adults who have confirmed CAP, should procalcitonin testing determine whether to initiate antibiotic therapy?
13	In adults with CAP, should the clinical prediction rule for prognosis – the Pneumonia Severity Index (PSI) – plus clinical judgement be used to determine the need for hospitalization, rather than clinical judgement alone?
14	In adults with CAP, should the clinical prediction rule for prognosis – the CURB-65 – plus clinical judgement be used to determine the need for hospitalization, rather than clinical judgement alone?
Antibiotic therapy
Outpatients without co-morbidities or risk factors for drug-resistant pathogens
15	Should outpatient adults with CAP who don’t have co-morbidities or risk factors for drug-resistant pathogens be treated empirically with amoxicillin monotherapy?
16	Should outpatient adults with CAP who don’t have co-morbidities or risk factors for drug-resistant pathogens be treated empirically with doxycycline monotherapy?
Outpatients with either co-morbidities or risk factors for drug-resistant pathogens
17	Should outpatient adults with CAP who have either co-morbidities or risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus macrolide combination therapy?
18	Should outpatient adults with CAP who have either co-morbidities or risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus doxycycline combination therapy?
19	Should outpatient adults with CAP who have either co-morbidities or risk factors for drug-resistant pathogens be treated empirically with respiratory fluroquinolone monotherapy?
Hospitalized adults with nonsevere CAP who have no risk factors for drug-resistant pathogens
20	Should hospitalized adults with nonsevere CAP who have no risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus macrolide combination therapy?
21	Should hospitalized adults with nonsevere CAP who have no risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus doxycycline combination therapy?
22	Should hospitalized adults with nonsevere CAP who have no risk factors for drug-resistant pathogens be treated empirically with respiratory fluroquinolone monotherapy?
Hospitalized adults with severe CAP who have no risk factors for drug-resistant pathogens
23	Should hospitalized adults with severe CAP who have no risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus macrolide combination therapy?
24	Should hospitalized adults with severe CAP who have no risk factors for drug-resistant pathogens be treated empirically with beta-lactam plus respiratory fluroquinolone combination therapy?
Other antibiotic questions
25	Should hospitalized adults with nonsevere CAP who have suspected aspiration pneumonia but no evidence of lung abscess or empyema be routinely treated with the addition of empiric anaerobic coverage?
Corticosteroids
26	Should hospitalized adults with nonsevere CAP be treated with corticosteroids?
27	Should hospitalized adults with severe CAP due to influenza be treated with corticosteroids?
Influenza
28	Should hospitalized adults with CAP who test positive for influenza be prescribed anti-influenza treatment (ie, oseltamivir)?
29	Should outpatient adults with CAP who test positive for influenza be prescribed anti-influenza treatment (ie, oseltamivir)?
30	Should outpatient or hospitalized adults with CAP who test positive for influenza be prescribed anti-bacterial antibiotics?
Follow-up
31	Should adults with CAP whose symptoms resolve within 5 days undergo routine follow-up chest radiography?

Outcomes Measures

A threshold of 70% or more of respondents selecting the same course of action was used during the CORE process to make a consensus recommendation. In other words, 70% or more of the respondents must select either a strong or a conditional recommendation for a given course of action for a recommendation to be made for that course of action. The 70% agreement threshold was chosen because it was found to be the optimal threshold in the original study of the CORE process [3].

CORE-derived recommendations were compared with recommendations from the IOM-adherent CAP guidelines following completion of the guideline. The proportion of questions that yielded recommendations was calculated, followed by concordance of the recommendations. “Concordance” is the number of CORE-derived recommendations that were identical to recommendations in the IOM-adherent guidelines, divided by the total number of CORE-derived recommendations. For example, if 12 questions yielded 10 CORE-derived recommendations, and 9 of the recommendations were identical to recommendations from the CAP guidelines, then concordance was 90%. Concordance of directionality (for or against the course of action) was initially determined. Then, among recommendations with the same directionality, concordance of the strength of the recommendation and the quality of evidence were also determined.

Statistical Analysis

To quantify agreement between recommendations derived from the CORE process and recommendations from the IOM-adherent CAP guidelines, a kappa coefficient was calculated [11]. A kappa coefficient is a measure of the magnitude of agreement between observers; in other words, how likely it is that the observers truly agree versus agree by chance. A kappa coefficient of one indicates perfect agreement, whereas a kappa coefficient of zero indicates that agreement is equally likely to be due to chance. For kappa coefficients values between zero and one, we defined 0.01–0.20 as slight agreement, 0.21–0.40 as fair agreement, 0.41–0.60 as moderate agreement, 0.61–0.80 as substantial agreement, and 0.81–1.00 as almost perfect agreement.

RESULTS

Twenty-six CAP experts were invited to participate in the CORE process, among whom 19 accepted, 3 declined, and 4 did not respond. All 19 (100%) individuals who accepted the invitation completed the portion of the CORE process that consisted of choosing the course of action, strength of recommendation, and quality of evidence (Supplementary Table 1). Only some participants provided free-text comments (Supplementary Table 2). The CAP guidelines addressed 33 questions in 16 categories; responses to 31 questions were analyzed (Table 1). Two questions were excluded from analysis because the complexity of the guideline recommendations was not amenable to comparison (the guideline made multiple recommendations addressing different subpopulations of the original population; therefore, CORE-derived responses to the original question were not comparable).

Using a threshold of 70% or more of respondents selecting the same course of action to make a recommendation, the CORE process yielded a recommendation for 20 of 31 (65%) questions. Among the 20 recommendations, 19 (95%) were concordant with the course of action recommended by the CAP guidelines (kappa agreement 0.88, 95% CI .64–1.00) (Table 2). Seven of the 11 (64%) questions for which there was insufficient agreement to make a recommendation addressed the choice of antibiotic regimen. This included questions about antibiotics for outpatients with and without comorbidities, hospitalized patients with nonsevere CAP, and hospitalized patients with severe CAP.

Table 2.

Directionality of Recommendations (for/against)

PICO Question	Modified CORE process Agreement	Modified CORE process Recommendation	IOM-adherent process Recommendation
		Consensus threshold of 70%	Consensus by discussion
1	3/19 (16%) for 16/19 (84%) against 0/19 (0%) no recommendation	Against	Against
2	18/19 (95%) for 1/19 (5%) against 0/19 (0%) no recommendation	For	For
3	0/19 (0%) for 19/19 (100%) against 0/19 (0%) no recommendation	Against	Against
4	14/19 (74%) for 5/19 (26%) against 0/19 (0%) no recommendation	For	Against
5	19/19 (100%) for 0/19 (0%) against 0/19 (0%) no recommendation	For	For
6	11/19 (58%) for 8/19 (42%) against 0/19 (0%) no recommendation	No recommendation	Against
7	17/19 (89%) for 2/19 (11%) against 0/19 (0%) no recommendation	For	For
8	9/19 (47%) for 10/19 (53%) against 0/19 (0%) no recommendation	No recommendation	Against
9	19/19 (100%) for 0/19 (0%) against 0/19 (0%) no recommendation	For	For
10	17/19 (89%) for 2/19 (11%) against 0/19 (0%) no recommendation	For	For
11	18/19 (95%) for 1/19 (5%) against 0/19 (0%) no recommendation	For	For
12	0/19 (0%) for 19/19 (100%) against 0/19 (0%) no recommendation	Against	Against
13	15/19 (79%) for 4/19 (21%) against 0/19 (0%) no recommendation	For	For
14	16/19 (84%) for 3/19 (16%) against 0/19 (0%) no recommendation	For	For
15	9/19 (47%) for 8/19 (42%) against 2/19 (11%) no recommendation	No recommendation	For
16	11/19 (58%) for 7/19 (37%) against 1/19 (5%) no recommendation	No recommendation	For
17	12/19 (63%) for 4/19 (21%) against 3/19 (16%) no recommendation	No recommendation	For
18	10/19 (53%) for 6/19 (32%) against 3/19 (15%) no recommendation	No recommendation	For
19	14/19 (74%) for 4/19 (21%) against 1/19 (5%) no recommendation	For	For
20	18/19 (95%) for 1/19 (5%) against 0/19 (0%) no recommendation	For	For
21	10/19 (53%) for 9/19 (47%) against 0/19 (0%) no recommendation	No recommendation	For
22	17/19 (89%) for 2/19 (11%) against 0/19 (0%) no recommendation	For	For
23	19/19 (100%) for 0/19 (0%) against 0/19 (0%) no recommendation	For	For
24	13/19 (68%) for 6/19 (32%) against 0/19 (0%) no recommendation	No recommendation	For
25	8/19 (42%) for 11/19 (58%) against 0/19 (0%) no recommendation	No recommendation	Against
26	0/19 (0%) for 18/19 (95%) against 1/19 (5%) no recommendation	Against	Against
27	3/19 (16%) for 16/19 (84%) against 0/19 (0%) no recommendation	Against	Against
28	19/19 (100%) for 0/19 (0%) against 0/19 (0%) no recommendation	For	For
29	18/19 (95%) for 1/19 (5%) against 0/19 (0%) no recommendation	For	For
30	10/19 (53%) for 8/19 (42%) against 1/19 (5%) no recommendation	No recommendation	For
31	7/19 (37%) for 11/19 (58%) against 1/19 (5%) no recommendation	No recommendation	Against
	Recommendations made: 20/31 (65%)    No recommendation, systematic review required: 11/31 (35%)    Concordance among recommendations yielded by the CORE process: 19/20 (95%)    Kappa agreement coefficient  a  : 0.88, 95% CI .64–1.00

^a 0.01–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, 0.81–1.00 near perfect agreement.

The lone discordant recommendation involved whether to routinely obtain blood cultures in hospitalized adults with nonsevere CAP who did not have risk factors for drug-resistant pathogens. The CORE process yielded a conditional recommendation for routine blood cultures, whereas the CAP guidelines made a conditional recommendation against routine blood cultures. Notably, the CAP guidelines recommendation was informed by very low-quality evidence and differed from the recommendation in the previous version of the guideline, despite there being no new empirical evidence since the previous version of the guideline [12]. If only one more of the 19 participants had chosen against blood cultures instead of for blood cultures, the CORE process would not have yielded a recommendation.

Among recommendations with concordant courses of action, 11 of 19 (58%) had a concordant strength of recommendation, and 8 of 19 (42%) had a discordant strength of recommendation (kappa agreement 0.03, 95% CI −.43 to +.48) (Table 3, Supplementary Table 3). The modified CORE process overestimated the strength for 4 of 8 (50%) and underestimated the strength for the remaining 4 of 8 (50%). Similarly, among recommendations with concordant courses of action, 8 of 19 (42%) had a concordant quality of evidence and 11 of 19 (58%) had a discordant quality of evidence (kappa agreement 0.13, 95% CI −.11 to +.37). The modified CORE process overestimated the quality of evidence for 7 of 11 (64%) and underestimated the quality of evidence for 4 of 11 (36%).

Table 3.

Strength of Recommendations (Strong/Conditional) and Quality of Evidence

Strength of Recommendations
Agreement in strength among concordant graded recommendations	11/19 (58%)
Overestimated strength	4/8 (50%)
Underestimated strength	4/8 (50%)
Kappa agreement coefficient for strength of recommendationa: 0.03, 95% CI −.43 to +.48
Quality of Evidence
Agreement in strength among concordant graded recommendations	8/19 (42%)
Overestimated strength	7/11 (64%)
Underestimated strength	4/11 (36%)
Kappa agreement coefficient for quality of evidencea: 0.13, 95% CI −.11 to +.37

^a0.01–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, 0.81–1.00 near perfect agreement.

The 70% threshold to make a recommendation was determined by prior work [3]. If the CORE process used a lower threshold of 60%, 2 additional recommendations would have been made, both of which would have been concordant with the CAP guidelines (ie, 22 of 31 [71%] questions would have yielded a recommendation with 21 of 22 [95%] concordance). Conversely, if CORE process used a higher threshold of 80%, 3 fewer recommendations would have been made, including the recommendation that was discordant with the CAP guidelines (ie, 17 of 31 [55%] questions would have yielded a recommendation with no discordant recommendations).

The time to develop the CAP guidelines from the panel’s first meeting until submission for peer review was approximately 4 years. In contrast, the time to complete the CORE process from distribution of the survey to collation and analysis of results was 3 weeks.

DISCUSSION

The purpose of the study was to estimate the impact of the CORE process if it had been utilized in the development of the ATS/IDSA Diagnosis and Treatment of Adults with CAP guidelines [1]. The results suggest that if the CORE process had been used in development of the CAP guidelines, 65% of the systematic reviews could have been avoided, assuming all CORE-derived recommendations were maintained following guideline panel discussion (Figure 1). That is, only 11 systematic reviews would have been necessary instead of the 31 that were conducted. As a result, using the CORE process would have saved effort and time [4]. In return, additional questions could have been addressed by the guideline panel within the same time frame and/or budget, providing much needed expert advice to the clinical community, rather than having to defer such advice to future iterations of the guideline. In addition, since the methodological demands of performing multiple systematic reviews can inhibit participation in guideline projects, a relatively agile methodology may be a valuable tool for addressing some clinical problems with similar results, particularly questions for which there is an anticipated high likelihood of expert agreement.

Figure 1.

Convergence of Opinion on Recommendations and Evidence (CORE) process. Abbreviation: PICO, population, intervention, comparator, outcome. Reprinted with permission of the American Thoracic Society. Copyright © 2020 American Thoracic Society. All rights reserved [4].

The impact of the CORE process on the CAP guideline’s final recommendations would have been minor. Among the 20 recommendations made by the modified CORE process, 19 (95%) would have advocated the same course of action. The only difference is that the CORE process would have yielded a recommendation for blood cultures in patients hospitalized with nonsevere CAP and no risk factors for drug-resistant pathogens, whereas the CAP guidelines recommended against blood cultures in such patients. We suspect this discordance reflects that different individuals participated in the guideline than in the CORE process, rather than being due to the different methodologies (i.e, the CORE process versus the IOM-adherent approach). The question was one of the most debated within the guideline panel, with the recommendation changing from a previous version of the guideline despite little change in the body of evidence.The CORE process has been criticized, largely based upon the “slippery slope” fallacy. Namely, concern that guideline developers will find the CORE process so convenient that they will utilize it to develop all recommendations, and systematic reviews will become extinct [13–15]. We emphasize this is NOT the intended use of the CORE process. The CORE process is a tool to select questions that require a systematic review and to efficiently formulate trustworthy recommendations for questions that do not require a systematic review. We believe that systematic reviews are an important tool for guideline developers; but we also believe that many systematic reviews performed during guideline development are unnecessary.

Best practice statements (ie, “motherhood” statements) are widely accepted in guideline development [16]. For a recommendation to be a best practice statement, it must be apparent that the alternative course of action is unreasonable. For example, “we recommend not dropping the baby” would be a best practice statement because a recommendation to drop the baby is an unreasonable alternative. Best practice statements do not require a systematic review. However, it is standard practice to perform a systematic review to inform all other recommendations. The CORE process is based upon the notion that for some questions for which systematic reviews are currently performed, the appropriate course of action is obvious to the expert panel and therefore, systematic reviews do not need to be performed (Figure 2). It has been argued that such questions should not be addressed in a guideline; in our opinion it is reasonable to address such questions in a guideline because, while obvious to the expert panel, they may not be obvious to the target audience. The CORE process is a methodology that could be utilized to address some clinical questions, while other questions may be more appropriately addressed with full guideline methodology, including conducting full systematic reviews to inform recommendations. Like best practice statements, CORE-derived recommendations should not be graded. Although studies consistently show that the CORE process yields recommendations whose course of action agree with recommendations developed using IOM-adherent guideline methodology, they have shown poor agreement in the strength of recommendation and quality of evidence derived from the two approaches.

Figure 2.

CORE process conceptual model. Abbreviation: CORE, convergence of opinion on recommendations and evidence. Best practice statements (ie, “motherhood” statements) are recommendations in which the appropriate course of action is obvious and, therefore, do not require a systematic review. The CORE process is based upon the belief that there exist questions for which the appropriate course of action is obvious to experts, but not to the general medical community, and such questions can also be answered without a systematic review. These circumstances might occur in cases where a) the quality and quantity of evidence is high and consistent or b) the quality of evidence is expected to be poor and unlikely to yield meaningful results beyond expert opinion. In cases where the data are contradictory or uncertain, however, systematic reviews are more likely to be required to inform recommendations.

Our study has limitations. First, the scope of questions included only CAP and therefore, it is unknown if our findings can be extrapolated to other topics. Second, there are previous versions of the CAP guidelines and it cannot be determined how the prior guidelines may have influenced the decisions of the participants in the CORE process. Third, the experts who completed the CORE process were not members of the guideline panel. Thus, it cannot be determined if the lone discordant recommendation differed because of the participants, their experiences, or the different methodology of the two processes used. Fourth, the CAP guidelines participants were mostly from the United States, whereas the CORE participants were a more international group. The effect of this geographic difference in participants is unknown but could have biased the results toward discordance due to different practice patterns and microbiology. Fifth, only a single round survey was performed, so it is unknown if the results may have been different if an iterative process had been performed. In other studies, the CORE process was performed using 1 to 3 rounds of surveys. While a high degree of agreement has been observed with a single round survey, our belief is that 2 rounds are ideal. Our observation is that responses tend to change the first time the participants see their colleagues’ responses but remain relatively static thereafter. Sixth, the small number of participants means that the outcome could be influenced by 1 or 2 participants’ choices. Case in point, the lone discordant recommendation would have differed if a single participant changed their choice. The optimal number of participants is unknown, but it is reasonable to expect that the more participants the better, as more participants decreases the likelihood that a single response will affect the conclusion. Finally, the CAP guidelines panel had their industry relationships rigorously vetted and managed, whereas the CORE process participants did not. It cannot be determined if this may have contributed to the lack of consensus on some questions, particularly those on the choice of antibiotic regimens. Despite these limitations, the results support previous studies that reported 1) the CORE process identifies recommendations that do not need to be informed by a systematic review and 2) CORE-derived recommendations are concordant with recommendations informed by a systematic review of the evidence [3, 4].

It is our belief that concordance between CORE-derived recommendations and recommendations informed by a systematic review reflects the evolving definition of a “subject expert.” In the past, experts routinely made recommendations based upon biological rationale and clinical experience alone, with little regard for published evidence. Since then, the term “evidence-based medicine” was coined [17] and experts have become defined by their role in evidence generation and their knowledge of the evidence. As a result, systematic reviews are increasingly unlikely to uncover important studies that are unknown to an expert panel, yet the time, effort, and cost of doing the systematic reviews remain, and may be increasing as the literature is ever-expanding. We believe that systematic reviews are important tools in guideline development but can be performed selectively when expert panels are unable to reach consensus [3, 4].

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Author contributions. Conception and design: all authors. Data collection: K. C. W. Data analysis: all authors. Writing the manuscript: K. C. W. Editing the manuscript: all authors.

Acknowledgments. CAP Guideline Panelists: Antonio Anzueto, San Antonio, Texas, USA. Jan Brozek, Hamilton, Ontario, Canada. Laura Cooley, Atlanta, Georgia, USA. Kristina Crothers, Seattle, Washington, USA. Nathan Dean, Murray, Utah, USA. Michael Fine, Pittsburgh, Pennsylvania, USA. Scott Flanders, Ann Arbor, Michigan, USA. Marie Griffith, Nashville, Tennessee, USA. Ann Long, Seattle, Washington, USA. Mark Metersky, Farmington, Connecticut, USA. Joshua Metlay, Boston, Massachusetts, USA (IDSA co-chair). Daniel Musher, Houston, Texas, USA. Marcos Restrepo, San Antonio, Texas, USA. Grant Waterer, Perth, Australia (ATS co-chair). Cynthia Whitney, Atlanta, Georgia, USA. CAP experts not on the guideline panel: Robert Balk, Chicago, Ilinois, USA. Jean Chastre, Paris, France. Catia Cilloniz, Barcelona, Spain. Sushma Cribbs, Atlanta, Georgia, USA. Charles Dela Cruz, New Haven, Connecticut, USA. Scott Evans, Houston, Texas, USA. Charles Feldman, Johannesburg, South Africa. Matthew Goetz, Los Angeles, California, USA. Lauri Hicks, Atlanta, Georgia, USA. Barbara Jones, Salt Lake City, Utah, USA. Lionel Mandell, Hamilton, Ontario, Canada. Rosario Menendez, Valencia, Spain. Eric Mortensen, Farmington, Connecticut, USA. Lucy Palmer, Stony Brook, New York, USA. Paula Peyrani, Louisville, Kentucky, USA. Maria Rodriguez-Barradas, Houston, Texas, USA. Wesley Self, Nashville, Tennessee, USA. Antonio Torres, Barcelona, Spain. Richard Wunderink, Chicago, Illinois, USA.

Financial support. This study was supported by the American Thoracic Society (ATS) as a derivative project of the joint ATS and Infectious Diseases Society of America funded Clinical Practice Guideline on the Diagnosis and Treatment of Community-acquired Pneumonia.

Potential  conflicts of interest. K. C. W. and N. C. S. developed the Convergence of Opinion on Recommendations and Evidence (CORE) process; neither is positioned to benefit financially from the CORE process. K. C. W. reports being the ATS Documents Editor and the ATS Chief of Guidelines and Documents. As a result, he has a professional interest in the quality of all ATS clinical practice guidelines. D. L. C., K. C., K. P. F., J. P. M., J. J. S., C. S., G. W., and R. D. have no conflicts to disclose. Both authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.