Evidence-Based Medicine and the American Thoracic Society Clinical Practice Guidelines

Abstract

This study assesses whether American Thoracic Society clinical practice guidelines are substantiated by high-quality evidence and can be used to promote evidence-based medicine.

The American Thoracic Society (ATS) issues clinical practice guidelines for the care of patients with pulmonary and critical care disease. The utility of ATS guidelines depends on the quality of the evidence base underpinning recommendations and whether the guidelines permit the practice of evidence-based medicine (EBM).^1,2 However, the extent to which ATS guidelines are substantiated by high-quality evidence and can be used to promote EBM is unknown.

Methods

Two of 3 investigators (R.C.S., K.D., and A.N.M.) reviewed each ATS clinical practice guideline recommendations listed on the ATS website as of August 1, 2017, that pertained to adults.³ We abstracted the following domains necessary for evidence-based clinical decision making based on prior conceptual frameworks^1,2: recommendation type, recommendation strength (using the Grading of Recommendations Assessment, Development, and Evaluation [GRADE] scoring system⁴ of strong [benefits clearly outweigh risks in most patients] vs low/conditional [benefits do not clearly outweigh risks in a substantial minority of patients]), quality of evidence (using GRADE categories⁴ of high [further research is unlikely to change estimate of effect] to very low [any estimate of effect is uncertain]), EBM measures, and patient context. Institutional review board approval was not needed because no human participants were included.

We defined recommendations as including the basic set of EBM measures if they included at least 1 measure of test performance for diagnostic recommendations (sensitivity, specificity, or likelihood ratio) and at least 1 measure of absolute benefit or harm for therapeutic recommendations (absolute risk reduction/increase, number needed to treat/harm, or relative risk with incidence of the outcome for the control group). For patient context, we ascertained whether the narrative text included any discussion of a person’s severity of illness or comorbidities, sociopersonal context, prognosis, or personal preference and how these domains might influence the recommendation.¹ Differences between reviewers were resolved through negotiated consensus, aiming to achieve agreement using the most inclusive definitions. Two-sided P < .05 for descriptive statistics indicated significance.

Results

Among 222 unique recommendations from 16 separate guidelines, 141 (63.5%) were based on low-quality evidence, whereas fewer than 1 in 10 (19 [8.6%]) were based on high-quality evidence (Table 1). Nonetheless, 86 (38.7%) were designated strong recommendations. Higher quality of evidence was associated with an increased probability of receiving a strong recommendation; 29 of 141 low-quality evidence recommendations (20.6%), 41 of 62 moderate-quality evidence recommendations (66.1%), and 16 of 19 high-quality evidence recommendations (84.2%) were strongly recommended (P < .001 for trend). However, most strong recommendations were not supported by high-quality evidence (16 of 86 [18.6%]).

Table 1. Summary of the Evidence Base for ATS Clinical Practice Guidelines.

Recommendation Type	Recommendations, No.	Recommendations, No. (%)
		Strengtha		Quality of Evidenceb			Meets Basic Definition
		Strong	Low/Conditional	High	Medium	Low	EBM Measures	Patient Context
Overallc	222	86 (38.7)	136 (61.3)	19 (8.6)	62 (27.9)	141 (63.5)	102 (45.9)	101 (45.5)
Diagnostic	52	19 (36.5)	33 (63.5)	1 (1.9)	16 (30.8)	35 (67.3)	26 (50.0)	3 (5.8)
Therapeutic	165	65 (39.4)	100 (60.6)	18 (10.9)	44 (26.7)	103 (62.4)	76 (46.1)	98 (59.4)

Abbreviations: ATS, American Thoracic Society; EBM, evidence-based medicine.

^aClassified using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) score except the guideline for community-acquired pneumonia (CAP), which used strong (most patients should receive the intervention), medium, and weak (many health care professionals would not follow this recommendation) categories. CAP recommendations of medium or weak were reclassified as low/conditional.

^bClassified using the 4 GRADE categories, combining low- and very-low-quality ratings into a single low category, given the similar uncertainty and because several ATS guidelines only used 1 of these categories.

^cIncludes diagnostic, therapeutic, screening (n = 3), and monitoring (n = 3) recommendations. One recommendation included both a therapeutic and diagnostic recommendation.

Of 52 diagnostic testing recommendations, 26 (50.0%) presented the test’s sensitivity, specificity, or likelihood ratios. Of 165 therapeutic recommendations, 76 (46.1%) reported the treatment’s absolute benefits or harms. Overall, 101 recommendations (45.5%) included any discussion of patient context, and this did not differ by recommendation strength (61 [44.9%] for low/conditional vs 40 [46.5%] for strong recommendations; P = .81) or quality of evidence (65 [46.1%] for low-, 29 [46.8%] for medium-, and 7 [36.8%] for high-quality recommendations; P = .73). The most commonly discussed domains were severity of illness or comorbidities (84 [37.8%]) and sociopersonal context (51 [23.0%]). Prognosis (14 [6.3%]) and patient preference (3 [1.4%]) were rarely discussed.

Guidelines varied considerably in number, evidence base, and strength of recommendations (Table 2). The community-acquired pneumonia guideline had the most recommendations (n = 47), had the highest proportion supported by high-quality evidence (15 [31.9%]), and accounted for 15 of 19 total recommendations supported by high-quality evidence.

Table 2. Summary of Evidence Base by Guideline.

Disease Guideline (Year)	Recommendations, No.	Recommendations, No. (%)
		Strength			Quality of Evidence			Meets Basic Definition
		Strong	Medium	Low	High	Medium	Low	EBM	Patient Context
Asthma, severe (2014)	9	1 (11.1)	0	8 (88.9)	0	0	9 (100)	4 (44.4)	3 (33.3)
Asthma, exercise-induced (2013)	15	8 (53.3)	0	7 (46.7)	2 (13.3)	9 (60.0)	4 (26.7)	11 (73.3)	3 (20.0)
Bronchiolitis obliterans (2014)	7	0	0	7 (100)	0	0	7 (100)	7 (100)	4 (57.1)
CAP treatment (2007)	47	20 (42.6)	23 (48.9)	4 (8.5)	15 (31.9)	12 (25.5)	20 (42.6)	5 (10.6)	22 (46.8)
COPD exacerbation (2017)	7	1 (14.3)	0	6 (85.7)	0	2 (28.6)	5 (71.4)	7 (100)	5 (71.4)
Stable COPD (2011)	9	6 (66.7)	0	3 (33.3)	0	8 (88.9)	1 (11.1)	0	2 (22.2)
Exhaled NO (2011)	11	7 (63.6)	0	4 (36.4)	0	5 (45.4)	6 (54.5)	2 (18.2)	0
HAP/VAP (2016)	45	19 (42.2)	0	26 (57.8)	0	7 (15.6)	38 (84.4)	13 (28.9)	28 (62.2)
ILD/IPF (2015)	10	4 (40.0)	0	6 (60.0)	0	4 (40.0)	6 (60.0)	9 (90.0)	6 (60.0)
LAM (2016)	5	2 (40.0)	0	3 (60.0)	0	2 (40.0)	3 (60.0)	3 (60.0)	0
pHTN in sickle cell (2011)	7	4 (57.1)	0	3 (42.9)	0	4 (57.1)	3 (42.8)	5 (71.4)	7 (100)
PE in pregnancy (2011)	8	3 (37.5)	0	5 (62.5)	0	0	8 (100)	1 (12.5)	0
Sleep apnea (2013)	4	1 (25.0)	0	3 (75.0)	0	1 (25.0)	3 (75.0)	0	3 (75.0)
TB diagnosis (2017)	20	5 (25.0)	0	15 (75.0)	0	4 (20.0)	16 (80.0)	17 (85.0)	1 (5.0)
TB treatment (2016)	14	5 (35.7)	0	9 (64.3)	2 (14.3)	3 (21.4)	9 (64.3)	14 (100)	14 (100)
Ventilator weaning (2017)	4	0	0	4 (100)	0	3 (75.0)	1 (25.0)	4 (100)	2 (50.0)

Abbreviations: CAP, community-acquired pneumonia; COPD, chronic obstructive pulmonary disease; EBM, evidence-based medicine; HAP/VAP, hospital-acquired pneumonia/ventilator-associated pneumonia; ILD/IPF, interstitial lung disease/idiopathic pulmonary fibrosis; LAM, lyphangioleiomyomatosis; NO, nitric oxide; pHTN, pulmonary hypertension; PE, pulmonary embolism; TB, tuberculosis.

Discussion

The ATS clinical practice guidelines are supported by suboptimal evidence and often are not presented in a manner suitable to optimize care for individual patients. Although 38.7% of recommendations are strong, fewer than 1 in 10 are supported by high-quality evidence (ie, a randomized controlled trial or a meta-analysis). Apart from community-acquired pneumonia, only 4 recommendations were supported by high-quality evidence among the 175 recommendations for all other pulmonary and critical care conditions combined.

The main limitation of our study was that we overestimated the frequency that recommendations included EBM measures and patient context domains, because we did not require meaningful and complete information in these areas. For example, if a recommendation mentioned consideration of costs, we credited the recommendation for including discussion of a patient’s sociopersonal context.

Our study has several implications. First, most ATS guidelines should be cautiously applied and should not be considered standard of care, given the paucity of high-quality evidence. Future randomized clinical trials could lead to reversal of many recommendations, even for tests and therapies that are currently strongly recommended.⁵ Further, standardizing poorly substantiated diagnostic tests or treatments can lead to inappropriate and harmful care.⁶ Second, future ATS guideline committees should strive to adhere to best practices for EBM, which include a transparent discussion of diagnostic test characteristics, treatment effectiveness, and relevant patient context domains to help guide clinical decision making.² Third, our framework can be used by guideline development checklists and methodology reports to ensure future guidelines adheres to EBM principles.