Rheumatology Informatics System for Effectiveness (RISE) Practices See Significant Gains in Rheumatoid Arthritis Quality Measures

Abstract

Objective

Using the American College of Rheumatology’s RISE registry, we examined performance on rheumatoid arthritis (RA) quality measures and assessed the association between practice characteristics and changes in performance over time among participating practices.

Methods

We analyzed data from practices enrolled in RISE between January 1, 2015 and December 31, 2017. Eight quality measures in the areas of RA disease management, cardiovascular risk reduction, and patient safety were examined. Variability in performance was evaluated at the practice level. Multivariate linear models were used to predict change in measure performance by year and determine the effect of practice characteristics on change in performance over time.

Results

Data from 59,986 patients from 54 practices was examined. Mean age was 62±14 years, 77% were female, 69% were Caucasian, and most patients were seen in a single-specialty group practice (46%). Average performance on measures related to RA treatments was consistently high (>90%) across the study period. Measures related to RA functional status and disease activity assessment had the greatest improvements over time (8.4% and 13.0% increase per year, respectively; p<0.001). Single-specialty group practices had the fastest rates of improvement over time across all measures.

Conclusions

Among practices participating in RISE between 2015 to 2017, performance on most RA quality measures improved. Single specialty group practices saw the fastest rates of improvement over time. Identification of workflow patterns leading to dramatic improvements in quality of care will help guide process redesign to address gaps in priority areas, such as TB screening and blood pressure control.

Rheumatology Informatics System for Effectiveness (RISE) is an electronic health record (EHR)-enabled registry developed by the American College of Rheumatology to facilitate quality improvement among rheumatology practices nationally. RISE passively extracts EHR data from individual practices, aggregates and analyzes these data centrally, and feeds this information back to clinicians as actionable data using a web-based quality dashboard. By providing robust health IT infrastructure, the registry aims to decrease the burden of data collection on practices and streamline participation in federal quality reporting programs such as the Merit-based Incentive Payment System (MIPS). An additional benefit of the dashboard is to facilitate local rapid-cycle quality improvement by providing continuous performance feedback and benchmarking (1).

Previous studies have shown gaps in RA care in many different settings, including timely initiation and maintenance of RA treatments (2, 3), patient safety (4) and cardiovascular risk reduction (5). However, these studies have been limited to single institutions, regions, or to administrative data from a single insurance carrier, thus lacking generalizability. In recent years, several new performance measures for RA have been endorsed that are now operationalized as EHR-enabled measures (e-measures). New e-measures such as disease activity and functional status assessments require changes in workflow, which can make implementation difficult (6, 7). Practices with long-established workflows may be more equipped to capture specific data elements than practices with more recent changes to clinical workflows. Existing studies did not report on new measures, nor on whether participation in the RISE registry, with access to a dashboard that facilitates quality improvement were associated with improvements in performance. In addition, although EHR-derived performance on quality measures has been previously reported (8), due to lack of interoperability, direct comparisons of performance on measures between different EHR systems has not been possible to date among rheumatology practices.

In this study we aim to examine performance on eight quality measures most relevant to the care of patients with rheumatoid arthritis and identify practice characteristics associated with high performance or substantial improvements in performance over a three-year period. We examine performance on measures for the subset of patients with rheumatoid arthritis (RA) since RA was the initial focus for quality improvement for the registry.

Patients and Methods

Study population and timelines

Data derived from the ACR’s RISE registry. RISE is a national, EHR-enabled registry that passively collects data on all patients seen by participating practices, reducing the selection bias present in single-insurer claims databases (9). As of December 2017, RISE held validated data from 1,257 providers in 236 practices, representing about 36% of the U.S. clinical rheumatology workforce. We analyzed data collected on all patients with prevalent diagnosis of RA between January 1, 2015 and December 31, 2017. For each quality measure, the measurement period was defined as the 12 months (24 months for tobacco use screening and cessation) preceding the last date of each quarter during which a visit occurred (e.g., if a patient with RA had an RA-coded visit on Dec 15, 2015, the measurement period was defined from January 1, 2015 to December 31, 2015).

RISE is a dynamic registry with practices able to enter and leave over time. To allow for consistent longitudinal analysis, we only included practices that were enrolled in RISE during the entire study period and had ≥ 30 RA patients at every quarter in order to reduce the variation in performance due to small sample sizes. Overall, 54 practices were included; 6 were excluded because they did not contribute data during the entire period and 4 were excluded because they did not have ≥30 RA patients at every quarter). We included patients who had at least one clinical face-to-face encounter in each quarter of the year, thus patients were not included in the denominator during quarters when they did not have any clinical face-to-face encounters. An RA diagnosis was defined as having two ICD9/10 codes for RA (714.0/M06.9, respectively) ≥ 30 days apart. For each patient we only included quality measures that were recorded at or after the first clinic visit associated with an ICD9/10 code for RA.

Quality measures

As of December 2017, the RISE registry calculated patient-level performance on 24 quality measures (complete list available in Supplementary material, Table 1). We examined quality measures in the areas of RA management including disease activity assessment, functional status assessment, and disease modifying anti-rheumatic drug (DMARD) prescribing. Cardiovascular risk reduction and patient safety measures, specifically tobacco use screening and cessation counseling, blood pressure control, tuberculosis (TB) screening prior to biologic drug start, and use of high-risk medication in the elderly were also examined. Performance on each measure was defined as detailed in Table 1. The use of high-risk medication in the elderly measures are reported to MIPS as inverse measures, with lower percentages indicating better performance. In order to pool performance across quality measures in this study, performance on these (inverted) measures was inverted such that higher percentages indicated better performance – e.g., a performance of 1% on the inverted measure became 99% in the modified measure. We selected these eight measures because they are relevant to the care of RA patients, endorsed by the National Quality Forum (NQF), and have been implemented in the RISE registry since January 2015.

Table 1.

RISE Quality Measures included in this study

Measure ID	NQF#	CMS#	Measure Title	Measure Definition	NQS Domain	Sub-specialty	Measure Type
ACR 01	2523	N/A	Disease Activity Measurement for Patients with Rheumatoid Arthritis (RA)	Percentage of patients 18 years and older with a diagnosis of rheumatoid arthritis whose disease activity is assessed using a standardized measurement tool at 50% or more encounters for RA with the same clinician during the measurement period.	Effective Clinical Care	Rheumatoid Arthritis (RA) Measures	Process
ACR 02	2524	N/A	Functional Status Assessment for Patients with Rheumatoid Arthritis (RA)	Percentage of patients 18 years and older with a diagnosis of rheumatoid arthritis whose functional status is assessed using a standardized measurement tool at least once during the measurement period.	Effective Clinical Care	Rheumatoid Arthritis (RA) Measures	Process
ACR 03	0054	N/A	Disease-Modifying Anti-Rheumatic Drug (DMARD) Therapy for Active Rheumatoid Arthritis (RA)	Percentage of patients 18 years and older with active rheumatoid arthritis who are treated with a disease-modifying anti-rheumatic drug (DMARD) during the measurement period.	Effective Clinical Care	Rheumatoid Arthritis (RA) Measures	Process
ACR 04	N/A	N/A	Tuberculosis (TB) Test Prior to First Course Biologic Therapy	Percentage of patients 18 years and older with a diagnosis of rheumatoid arthritis that are newly prescribed a biologic therapy during the measurement period and whose medical record indicates tuberculosis testing in the 12 months preceding the biologic prescription.	Patient Safety	TB Measures	Process
PQRS 226	0028	138v4	Preventive Care and Screening: Tobacco Use: Screening and Cessation Intervention	Percentage of patients aged 18 years and older who were screened for tobacco use one or more times within 24 months AND who received cessation counseling intervention if identified as a tobacco user	Community and Population Health		Process, Cross-cutting
PQRS 236	0018	165v4	Controlling High Blood Pressure	Percentage of patients 18–85 years of age who had a diagnosis of hypertension and whose blood pressure was adequately controlled (<140/90mmHg) during the measurement period.	Effective Clinical Care	Hypertension Measure	Intermediate Outcome, Cross-cutting
PQRS 238	0022	156v4	Use of High-Risk* Medications in the Elderly	Percentage of patients 66 years of age and older who were ordered high-risk medications. Two rates are reported. a. Percentage of patients who were ordered at least one high-risk medication. b. Percentage of patients who were ordered at least two different high-risk medications. INVERSE MEASURE: Lower count indicates better performance	Patient Safety		Process

The measurement period for all measures is 12 months, unless stated. Practice-level performance was calculated at every quarter. Patients included in the denominator at every quarter must have had at least one visit during that quarter.

^*List of medications defined as high-risk available at: https://qpp.cms.gov/docs/QPP_quality_measure_specifications/CQM-Measures/2019_Measure_238_MIPSCQM.pdf

Other variables

We examined both sociodemographic characteristics of patients as well as practice characteristics using RISE data extracted through June 30, 2018. Practice characteristics included number of providers, practice type (multi-specialty group practice, single-specialty group practice, solo practitioner, and health system), practice EHR software (NextGen, eClinicalWorks, GE Centricity, Allscripts, Amazing Charts, Greenway/Primesuite, eMD-Plus, and UniCharts) and U.S. geographical region (South, West, Northeast, Midwest). Practice-level sociodemographic information was calculated from patients eligible for each quality measure during each quarter and included mean age, proportion female, proportion non-Caucasian and proportion with public insurance (Medicare or Medicaid).

Statistical analysis

Practice-level performance on quality measures, defined as the percentage of eligible patients in a practice receiving recommended care, was the primary outcome. For each measure, we reported the median performance and performance at the 99^th percentile in each year (2015, 2016 and 2017). We assessed changes in performance over time by calculating the change in performance on each measure across practices and comparing within-practice changes in performance across geographic regions. Intraclass correlation coefficients (ICCs) were calculated to determine how much of the variability in each quality measure was explained by between-practice variability. Statistical process control charts were used to determine if changes in performance represented common-cause variation or improvement.

We used bivariate hierarchical linear mixed-effects models to predict change in practice-level measure performance per year, accounting for repeated measurement of practices over time(10). To determine the independent effect of practice characteristics on practice-level measure performance, we used multivariate models. All multivariate models adjusted for time (as a continuous predictor) and included number of providers in the practice, practice type, EHR software and U.S. region as predictors. We adjusted for U.S. region to account for residual confounding that may remain due to geographical variations in practice characteristics that were not available or the underlying prevalence of exposures or disease; for example, providers may be more likely to screen for TB in regions that have a high prevalence of this condition. Patient sociodemographic factors (mean age, proportion female, proportion non-Caucasian, and proportion with public insurance) varied across practices. To account for these differences, demographics of patients eligible for each quality measure at each practice were indirectly standardized to the overall RISE patient population eligible for the respective quality measure using standardized ratio weights (11, 12). Finally, we evaluated whether the change in measure performance over time was modified by any of the practice characteristics by fitting an interaction term between time and each of the covariates in separate multivariate models that included all practice predictors and accounted for repeated measurement of practices over time. All models were checked for linearity of continuous predictors using component plus residual plots and normality of the residuals using residual versus predictor and QQ plots. All analyses were performed in STATA V16.0. The study procedures were approved by the UCSF IRB.

Results

Data from 59,986 patients from 54 practices was examined. Mean age was 62±14 years, 77% were female, 69% were Caucasian, and 81% had private insurance. The most common practice structure was a single-specialty group practice (65%), followed by solo practitioner (18.5%) and multi-specialty group practice (15%). NextGen was the most commonly used EHR brand (63%), followed by eClinicalWorks (15%) and GE Centricity (7%). Most practices were in the South region of the U.S. (56%), followed by West (22%) and Northeast regions (13%) (Table 2).

Table 2.

Characteristics of patients and practices in the RISE registry.

Patient demographics, N = 59,986 a
Age mean (SD)	62.3 (13.9)
Female N (%)	46,117 (76.9)
Race/Ethnicity N (%)
Non-Hispanic Caucasian	41,126 (68.6)
Black or African American	5,409 (9.0)
Hispanic or Latino	2,650 (4.4)
Asian	893 (1.5)
Other	6,102 (10.2)
Missing	3,806 (6.3)
Insurance status N (%)
Private	48,402 (80.7)
Medicare ≥65	3,303 (5.5)
Medicaid	2,106 (3.5)
Medicare <65	1,318 (2.2)
None	8 (0)
Missing	4,849 (8.1)
Practice characteristics	Patient-level, N = 59,986	Practice-level, N = 54 b
Practice size, N (%)
1–4 providers	23,023 (38.4)	32 (59.3)
5–9 providers	19,805 (33.0)	15 (27.8)
10–20 providers	17,158 (28.6)	7 (13.0)
Practice type, N (%)
Single-Specialty Group Practice	45,018 (75.1)	35 (64.8)
Solo Practitioner	4,837 (8.1)	10 (18.5)
Multi-Specialty Group Practice	9,613 (16.0)	8 (14.8)
Health System	518 (0.9)	1 (1.9)
EHR software, N (%)
NextGen	40,969 (68.3)	34 (63.0)
eClinicalWorks	12,065 (20.1)	8 (14.8)
GE Centricity	2,753 (4.6)	4 (7.4)
Allscripts	2,631 (4.4)	3 (5.6)
Amazing Charts	886 (1.5)	2 (3.7)
Otherc	682 (1.1)	3 (5.6)
U.S. geographic region
South	38,879 (64.8)	30 (55.6)
West	5,824 (9.7)	12 (22.2)
Northeast	6,076 (10.1)	7 (13.0)
Midwest	9,207 (15.4)	5 (9.3)

^aDynamic cohort

^bFixed cohort (practices that remained in RISE from Jan 2015 to Dec 2017)

^cOther included Greenway/Primesuite, e-MD-Plus and UniCharts

SD: Standard deviation; IQR: Inter quartile range.

We found a large amount of variability in performance on these measures over the three-year study period observing high-performing practices and practices with substantial improvements over time (Table 3). Performance on disease activity assessment, functional status assessment, TB screening and blood pressure control had the highest variability across practices with percentage point changes over the 3 years ranging from −97% to +99%; as a result, these quality measures were selected for further multivariate analysis. Median performance on tobacco screening and cessation was high (>80%) and saw greatest improvements during the earlier parts of the study period. Average performance on DMARD prescribing, use of one high risk medication and use of ≥2 high risk medications in the elderly was also consistently high (>90%) (Figure 1; control charts for all measures available in the supplementary material). We assessed change in performance over time on disease activity assessment, functional status assessment, TB screening and blood pressure control across U.S. regions. Most practices in the Northeast region saw no change or decreases in average performance on these four measures while practices in the West had improvements in performance (Figure 2). We also observed few practices within the South with substantial improvements in performance over time.

Table 3.

Variability in practice-level performance and change in practice-level performance across measures

	2015		2016		2017
Quality measure	Number of RA patients eligible for measure	Practice level performance, percentile (50th, 99th)	Number of RA patients eligible for measure	Practice level performance, percentile (50th, 99th)	Number of RA patients eligible for measure	Practice level performance, percentile (50th, 99th)	Percentage point change in performance from 2015 to 2017, range (min, max)
Disease activity assessment	36,355	57, 100	32,344	68, 100	29,803	78, 100	−97, 98
Functional status assessment	36,355	56, 100	32,344	77, 100	29,803	85, 100	−59, 99
DMARD prescribing	36,355	95, 100	32,609	96, 99	29,803	95, 100	−21, 59
TB screening	4,680	67, 100	2,427	71, 100	1,660	67, 100	−72, 74
Tobacco screening and cessation	36,989	91, 100	33,331	91, 100	30,497	92, 100	−32, 85
Blood pressure control	4,889	63, 100	6,512	63, 93	6,924	58, 100	−58, 46
One high-risk medication in elderly	15,910	97, 100	15,170	97, 100	15,207	97, 100	−12, 82
Two high-risk medications in elderly	15,910	100, 100	15,170	100, 100	15,207	100, 100	−1, 1

Performance on quality measures was defined as the percentage of eligible patients within practice receiving recommended care.

Figure 1.

Proportion of patients with rheumatoid arthritis meeting the quality measures from 2015 to 2017

Figure 2.

Within-practice change in average measure performance from 2015 to 2017 across the U.S. regions

Between-practice variability explained about half of the variation in performance on quality measures across the study period (ICCs ranged from 41% for tobacco use screening and cessation to 58% for TB screening) indicating important within-practice changes in performance over time. Results from bivariate hierarchical linear mixed-effects models (predicting change in performance as a function of time) from January 2015 to December 2017 showed there was a significant improvement in performance on functional status assessment (+13.9% per year; 95% CI: +11.8%, +16%; p<0.001), and disease activity assessment (+8.4%; 95% CI: +6.2%, +10.5%; p<0.001).There were smaller improvements in performance on TB screening (+4.3%; 95% CI: +2.8%, +5.7%, p<0.001) and tobacco screening and cessation (+2.9%; 95% CI: +1.8%, +4%, p<0.001). While improvements in blood pressure control (+1.6%; 95%CI: +0.2%, +3%, p=.022) and DMARD prescribing (+1%, 95%CI: +0.3%, +1.6%, p=.004) were statistically significant, they can most reliably be explained by expected common-cause variation (Supplementary material, Figures 3 and 6). Changes in performance on the use of high-risk medication in the elderly over time were negligible and not statistically significant.

Multivariate analyses showed that at any timepoint, larger practices with 10–20 providers performed better than small practices with 1–4 providers on all four measures (disease activity assessment, functional status assessment, TB screening and blood pressure control) with differences reaching statistical significance for functional status assessment (75.5% versus 45.0%, p=0.001) and blood pressure control (71.5% versus 59.7%, p=0.001) (Supplementary material, Table 2). Performance on disease activity assessment and functional status assessment was significantly higher in health systems compared to single specialty group practices. Northeast region practices had better performance than those in the South on TB screening and blood pressure control but demonstrated worse performance on disease activity assessment.

For each of the four quality measures, we also determined the effect of practice characteristics on change in measure-performance per year. Single specialty group practices had significantly higher rates of improvement per year than health systems, across all four measures (p for interaction ≤0.010) (Table 4). Single specialty group practices also had higher gains in performance than multi-specialty group practices across measures, although differences did not reach statistical significance. The EHR software eClinicalWorks was associated with faster improvements in functional status assessment than NextGen. NextGen was associated with faster improvements in disease activity assessment than both Allscripts and Amazing Charts. The West was associated with significantly faster improvements in TB screening than the South.

Table 4.

The effect of practice characteristics on change in measure performance per year

	Disease activity		Functional status		Tuberculosis screening		Blood pressure control
	Change in Performance % (95% CIs)	p-value	Change in Performance % (95% CIs)	p-value	Change in Performance % (95% CIs)	p-value	Change in Performance % (95% CIs)	p-value
Number of providers
1–4	5.3 (−0.9, 11.5)	REF	14 (8.2, 19.8)	REF	5.2 (0.2, 10.3)	REF	1.7 (−2.4, 5.8)	REF
5–9	13.4 (3.3, 23.6)	0.178	12.4 (−0.1, 24.9)	0.823	3.3 (−1.9, 8.4)	0.586	0.8 (−1.8, 3.3)	0.709
10–20	10.1 (0.2, 20)	0.420	18.2 (7.6, 28.8)	0.490	2.1 (−5.3, 9.6)	0.493	−0.2 (−2.3, 1.9)	0.415
Practice type
Single-specialty group practice	13 (6.8, 19.1)	REF	17.7 (10.9, 24.5)	REF	4.7 (0.6, 8.9)	REF	1.7 (−1.7, 5)	REF
Solo practitioner	0.8 (−9.9, 11.4)	0.052	8.2 (−0.9, 17.3)	0.097	9.2 (0.7, 17.7)	0.348	2.3 (−2.4, 6.9)	0.835
Multi-specialty group practice	0 (−4.7, 4.6)	0.001	7.4 (−1.8, 16.7)	0.078	−.2 (−6.9, 6.5)	0.217	−1.5 (−7.8, 4.9)	0.386
Health system	−15.4 (−35.1, 4.3)	<0.001	−0.2 (−15.7, 15.4)	<0.001	−17.1 (−26, −8.1)	<0.001	−2.8 (−7, 1.4)	0.010
EHR brand
NextGen	9.2 (3.9, 14.5)	REF	12.5 (5.8, 19.2)	REF	3.9 (−0.3, 8.2)	REF	1.5 (−1.8, 4.8)	REF
eClinicalWorks	12.6 (1.1, 24.2)	0.587	25.3 (14.8, 35.9)	0.044	7.3 (−0.5, 15.2)	0.446	1 (−4.4, 6.5)	0.881
GE Centricity	13.7 (−11.5, 38.9)	0.726	17.6 (−2.6, 37.7)	0.633	−1.4 (−14.6, 11.9)	0.446	1.4 (−2.1, 4.8)	0.953
Allscripts	−3.2 (−9.6, 3.2)	0.004	−2.2 (−6.4, 2)	<0.001	−.4 (−9.1, 8.3)	0.372	2 (−1.8, 5.7)	0.858
Amazing Charts	−20.2 (−39.2, −1.2)	0.004	10.4 (−6.9, 27.7)	0.820	−1.2 (−8.5, 6)	0.225	−13.3 (−24.7, −1.8)	0.016
Other	7.4 (−25.5, 40.3)	0.916	16.6 (7.3, 25.8)	0.479	20.2 (8.6, 31.8)	0.011	8.6 (−3.6, 20.8)	0.265
Regions
South	8 (1.9, 14.1)	REF	12.5 (5.7, 19.4)	REF	1.4 (−2.4, 5.2)	REF	−0.3 (−3.7, 3.1)	REF
West	10.9 (−2.7, 24.5)	0.699	21.4 (10.8, 32)	0.163	10 (3, 17)	0.034	5.4 (−0.3, 11)	0.090
Northeast	2.3 (−8.1, 12.7)	0.348	4.9 (−7, 16.7)	0.267	2.7 (−9.3, 14.6)	0.845	−1.4 (−5.4, 2.5)	0.662
Midwest	9.3 (−0.1, 18.8)	0.818	16.2 (2.3, 30.1)	0.637	10 (−4.4, 24.5)	0.254	3.5 (−0.6, 7.5)	0.157

Marginal means estimated using weighted multivariate regression models accounting for repeated measurement of practices over time. The model additionally incorporated year as a continuous variable. P-values indicate statistical significance for interaction.

Discussion

Practices participating in the RISE registry had significant improvements over time in performance on multiple quality measures, including RA disease activity assessment, functional status assessment, TB screening, and tobacco screening and cessation. The greatest improvements were in the assessment of functional status and disease activity. We observed considerable variability in performance across practices and regions. Larger practices had better performance on measures compared with small practices, while single specialty group practices had significantly faster rates of improvement over time compared with multispecialty group practices and health systems.

Two nationally endorsed RA-specific quality measures (disease activity assessment and functional status assessment) are the first examples of e-measures that collect outcomes, including patient reported outcomes, across the registry. We are encouraged to observe steady and significant improvements in performance on these measures. Measurement of these outcomes using validated tools facilitates a treat-to-target approach and is a key part of high-quality rheumatology care. Additionally, collection of these measures allows for tracking of outcomes, benchmarking across rheumatology practices and creation of a learning health system in which information about outcomes and performance is fed back to providers to continuously improve quality of care (13). Average performance on DMARD prescribing was consistently high (>90%) from January 2015 to December 2017. This is consistent with results from an earlier analysis of this measure using EHR data (94%) (8). Since performance on this measure has been shown to be optimized, this measure has been retired from the MIPS program.

Regarding measures related to cardiovascular disease prevention, performance on the blood pressure control measure in this study was suboptimal and comparable with previous reports (5). Suboptimal performance on other cardiovascular risk reduction measures have also been reported among RA patients in the U.S. and Canada, including hyperlipidemia and diabetes screening (14–16). Protocols for controlling hypertension are proven to be effective in primary care settings (17). Such protocols have not been extensively studied in specialty settings such as rheumatology clinics, but they hold potential. A recent study on the implementation of a rheumatology-primary care partnership protocol for the management of high blood pressure showed that timely primary care follow-ups for patients with in-network primary care reduced rheumatology visits by patients with high blood pressure indicating reduced population-level rates of high BP (18). Performance on tobacco use screening and cessation in this study was significantly higher than indicated from a previous study using manually abstracted data at an academic rheumatology practice (smoking status was documented at 39% of visit notes with smokers and smoking cessation counseling was documented in 10%) (19). Incompleteness of manual chart review abstraction and incomplete documentation in notes could explain part of the discrepancy. Notably, tobacco screening and cessation was part of the Meaningful Use program implemented across specialties in 2014; this may explain the steady increase in performance during the earlier parts of the study period.

Among patient safety measures, while performance on TB screening has improved since early 2015, the evident lower performance on this measure indicates both a gap in quality and the fact that reliably capturing TB screening in practice requires further work to ensure accurate data capture from the EHR. Low performance rates on the TB screening measure have been reported previously, even in studies that used extensive chart reviews to examine performance on this measure, and look-back periods that were longer than 12 months to define incident users (4). Therefore, it is likely that the low performance on the TB measure in RISE represents a meaningful gap in care. However, since there is currently no clear evidence to guide appropriate look-back periods for TB testing, it is possible that low performance at least partially reflects clinical controversy about which patients need updated TB screening.

We observed better performance on quality measures among larger practices at all points in time, however, single-specialty group practices saw the fastest rates of improvements over time. Larger practices including health systems and multi-specialty group practices likely have more resources to invest in quality improvement activities and infrastructure such as workflows that facilitate the documentation of disease activity and functional status assessments. Another explanation might be the availability of structured fields within more mature EHR systems for documentation of disease activity and patient reported outcomes. Single-specialty group practices saw greater gains in performance over time, possibly because RISE provides IT infrastructure for quality improvement and also because they had lower performance at the start of the study period and hence more room for improvement. Practice and regional variability data from RISE facilitate identification of targets for quality improvement and education initiatives.

We were interested in exploring whether EHR software was associated with performance, but found that commonly used vendors such as NextGen, eClinicalWorks and GE Centricity were comparable across most quality measures. This finding suggests that practices can achieve high performance regardless of software, and also that current software does not seem to support high quality performance. In addition, practices that join RISE may be selected to have clinical workflows to electronically capture required information and the necessary health information technology support staff to build and test the quality measures locally. It is important to note that measures selected in this study were part of different incentive programs over time; for example, tobacco cessation counseling was a meaningful use measure and therefore less likely to have significant variation between EHR vendors. In contrast, rheumatology-specific measures, such as functional status or disease activity assessments are less likely to be supported uniformly by EHRs and therefore may be more prone to variability. Notably, EPIC, one of the largest market-share holders among EHR vendors, was not used by practices participating in RISE during the study period. This is likely due to a combination of factors, including the fact that academic centers are the main users of EPIC and faced either institutional or vendor-related barriers in contributing data to the RISE registry.

Our study has important strengths. This study is the first to report change in performance on quality measures over time across a large patient population with diverse geographical coverage across the nation and in RISE practices using different EHR systems. In addition, we used statistical methods that account for variability in sociodemographics across practices and produce reliable estimates generalizable to the overall sociodemographic populations represented within RISE. Limitations of this study include lack of a control group. Without a comparison group of practices who did not join the RISE registry, it is unclear how much of the improvements in performance on these measures over time are attributable to participation in the registry itself. It is also possible that this data may underestimate the care provided to patients since documentation within EHR may be inconsistent and non-structured information is difficult to query systematically. To enable a meaningful longitudinal analysis, we included practices that were early and sustained users of the RISE registry; however, these practices were more likely to serve privately insured patients compared to all practices currently participating in the registry (1). Finally, since ICD codes were used to identify RA diagnosis in many denominators, it is possible that performance on these measures has been underestimated in our study, as some patients may not have truly had RA. However, since these codes were assigned by rheumatologists, the positive predictive value may be higher than in other studies where codes could be assigned by any provider (20).

With the capacity of RISE to facilitate rapid cycle quality improvements for participating practices and the emerging payment reforms put into place by Medicare Access and CHIP Reauthorization Act (MACRA) of 2015, there is an urgent need to develop new measures to define value in rheumatology. The measures assessed in this study were process measures that assess the actions taken in the course of healthcare. In 2016, the ACR began development of a new outcome measure to measure the effects of these actions on health status using clinical data from the RISE registry (21). Understanding the scientific validity, feasibility, usefulness and intended and unintended consequences of quality measures also continues to be an important strategic goal of RISE. As more practices join RISE, larger studies will be powered to facilitate further subgroup analyses that identify target areas for quality improvement. Key questions regarding the role of sociodemographic factors, healthcare access and patient satisfaction remain and can serve as the focus of future research. Furthermore, given the variability in performance across RISE practices, further qualitative research is needed to better understand facilitators and barriers to improvement on these measures.

In summary, this paper provides a systematic benchmarking of the ACR quality measures using data from 54 practices participating in the RISE registry. Results from this study indicate excellent performance on DMARD prescribing and steady improvements in documentation of disease activity and functional status over a three-year period between 2015 to 2017. Blood pressure control and TB screening measures may deserve the most attention in performance improvement initiatives, although notable improvements on these measures were observed among some practices. Identification of workflow patterns leading to high performance or dramatic improvements in quality of care will help guide strategies to address gaps in priority areas.

Significance and Innovations.

• Among practices participating in RISE, from 2015 to 2017 performance on most measures for individuals with RA improved.

• There were significant variations in performance over time between practices, suggesting that future work to identify workflow patterns leading to high performance or to dramatic improvements in quality are warranted.

• Performance on quality measures across RISE practices provides a useful benchmark for rheumatologists seeking to improve quality in their practices.