The Evidence Basis for the American College of Rheumatology Practice Guidelines

Abstract

This study evaluated the quality of the evidence base and the strength or benefit to harm ratio of the recommendations in the American College of Rheumatology Clinical Practice Guidelines

The American College of Rheumatology (ACR) places a high priority on developing methodologically rigorous, evidence-based Clinical Practice Guidelines (CPGs). To assess the evidential foundation of rheumatology guidelines, we evaluated (1) the level (quality) of the evidence base and (2) the class (strength or benefit to harm ratio) of the recommendations in the ACR CPGs.

Methods

As of March 10, 2017, we obtained 8 publically available CPGs (https://www.rheumatology.org) that assessed evidence using 3 different methodologies: American College of Cardiology/American Heart Association (ACC/AHA) approach in 3, Grading of Recommendations and Assessment, Development, and Evaluation scoring system (GRADE) in 4, and University of Oxford Centre for Evidence-Based Medicine approach in one. To compare the guidelines using an identical methodology, we classified the level (quality) of evidence and class (strength) of the recommendations using the ACC/AHA method.

Working independently, 2 reviewers (A.D. and R.Z.) extracted the reported evidence level and class for each recommendation. When recommendations involved multiple population, intervention, comparator and outcome (PICO) questions, we weighted their contribution to the recommendation equally (eg, each of 4 PICO questions contributed one-fourth of the evidence). Occasionally, consensus involved assigning a recommendation to 2 categories (ie, contributing 0.5 to each). We summarized the distribution of recommendations by level and class using medians and interquartile range to weigh each guideline equally.

Results

Glucocorticoid-induced osteoporosis (GIOP), juvenile idiopathic arthritis (JIA), gout, lupus nephritis, osteoarthritis (OA), ankylosing spondylitis (SpA), polymyalgia rheumatica (PMR), and rheumatoid arthritis (RA) comprised the 8 guidelines, which involved 403 (ranging from 10 to 102 each) recommendations.

Four guidelines (OA, SpA, PMR, and RA) reported class and level in their 143 recommendations, and 4 (GIOP, JIA, gout, and lupus nephritis) reported only evidence level. Across guidelines (Table 1), over half of the 403 recommendations were classified as level C, one-fourth (93 recommendations) level A, and one-fifth (77 recommendations) level B. The proportion of level A varied from 58% with OA and 35% with GIOP to 2% with JIA and 10% with PMR.

Table 1. Guideline Recommendations by Level (Quality) of Evidence.

Guideline	No.	Year	Methodology	Level of Evidence, No. (%)a
				A	B	C
GIOP	37	2010	ACC/AHA	13.0 (35)	7.0 (19)	17.0 (46)
JIA	102	2011-2013b	Oxford	1.7 (2)	12.2 (12)	88.1 (86)
Gout	88	2012c	ACC/AHA	18.5 (21)	27.4 (31)	42.1 (48)
LN	33	2012	ACC/AHA	8.0 (24)	2.0 (6)	23.0 (70)
OA	60	2012	GRADE	35.0 (58)	10.0 (17)	15.0 (25)
SpA	38	2015	GRADE	11.0 (29)	7.5 (20)	19.5 (51)
PMR	10	2015	GRADE	1.0 (10)	4.5 (45)	4.5 (45)
RA	35	2015	GRADE	4.6 (13)	6.1 (17)	24.3 (69)
Total	403			92.8 (23)	76.7 (19)	233.5 (58)
Median % (IQR)				23.0 (12-30)	18.0 (15-23)	50.0 (46-70)

Abbreviations: ACC/AHA, American College of Cardiology/American Heart Association; GIOP, glucocorticoid-induced osteoporosis; GRADE, Grading of Recommendations and Assessment, Development, and Evaluation scoring system; IQR, interquartile range; JIA, juvenile idiopathic arthritis; LN, lupus nephritis; OA, osteoarthritis; Oxford, Oxford Centre for Evidence-Based Medicine; PMR, polymyalgia rheumatica; RA, rheumatoid arthritis; SpA, spondyloarthritis.

^aLevel A evidence to multiple randomized clinical trials (RCTs) or meta-analyses; level B to single RCT or nonrandomized studies; and level C to opinion of experts, case studies, or standard of care.

^bIncludes JIA guidelines of 2011 and focused 2013 update.

^cIncludes gout part 1 and part 2 guidelines.

Table 2 examines the relationship between the level of evidence and class of the recommendations. A median of 10% specified both level of evidence A and class I; 6%, level B and class I, and 8% level C and class I. The RA guideline had the highest percentage (16%) of such strong class I recommendations based on weak C evidence. The combination class II and level C evidence was the most common at a median of 30% across guidelines and comprised 50% of the RA recommendations.

Table 2. Guideline Recommendations by Class of Recommendation and Level of Evidence.

Guidelines (No.)	GRADE Methodology Year	Class of Recommendation – Level of Evidence, No. (%)a
		I-A	I-B	I-C	II-A	II-B	II-C	III-A	III-B	III-C
OA (60)	2012	10.0 (17)	0	3.0 (5)	19.0 (32)	9.0 (15)	9.0 (15)	6.0 (10)	1.0 (2)	3.0 (5)
SpA (38)	2015	4.0 (11)	1.0 (3)	2.0 (5)	67.0 (18)	3.8 (10)	14.5 (38)	0.3 (1)	2.7 (7)	3.0 (8)
PMR (10)	2015	0	2.0 (20)	1.0 (10)	1.0 (10)	1.3 (13)	2.3 (23)	0	1.3 (13)	1.3 (13)
RA (35)	2015	3.0 (9)	3.3 (9)	5.6 (16)	1.6 (5)	2.8 (8)	17.7 (50)	0	0	1.0 (3)
Total (143)		17.0 (12)	6.3 (4.4)	11.6 (8.1)	28.3 (19.7)	16.8 (11.7)	43.4 (30.3)	6.3 (4.4)	4.9 (3.4)	8.3 (5.7)
Median % (IQR)		10.0 (7-12)	6.0 (2-12)	8.0 (5-12)	14.0 (9-21)	11.0 (10-13)	30.0 (21-41)	0 (0-3)	4.0 (1-8)	6.0 (4-9)

Abbreviations: GRADE, Grading of Recommendations and Assessment, Development, and Evaluation scoring system; IQR, interquartile range; OA, osteoarthritis; PMR, polymyalgia rheumatica; RA, rheumatoid arthritis; SpA, spondyloarthritis.

^aClass I strength of recommendation was assigned when a given procedure or treatment should be performed or administered (benefit much greater than harm); class II when a procedure or treatment is reasonable or may be considered (benefit greater than or equal to harm); and class III when procedure or treatment is not effective or useful and may be harmful (harm or no benefit).

Discussion

Our findings suggest the evidence supporting ACR recommendations is limited, with more than 50% and occasionally 2 of every 3 recommendations classified as level C, that is, based on experts, standards of care, case reports or series. Of the 35 class I recommendations, only 17 were supported by level A evidence. Our analysis reflects the different methodologies used in the ACR guidelines at 1 time point. Most recently, the ACR applies GRADE to evaluate the study design and the quality of the evidence.

The ACR recommendations overall remain mostly expert-based but are comparable with guidelines in other subspecialties. The findings underline the need to enhance the evidence level. Although RCT evidence level A for all clinical questions is infeasible, in this era of “big data,” the creation of registries, (eg, ACR’s RISE and federal real-world data initiatives) should increase opportunities to build a learning health care system. Such a system would identify heterogeneity and gaps in care practices and outcomes, which would help set research priorities that would enhance the quality of rheumatology care and increase appropriate use and patient-centered clinical outcomes while minimizing unwarranted practice variation.