Effectiveness of a multistate quality improvement campaign in reducing risk of surgical site infections following hip and knee arthroplasty

Abstract

Background:

Quality improvement (QI) campaigns appear to increase use of evidence-based practices, but their effect on health outcomes is less well studied.

Objective:

To assess the effect of a multi-state QI campaign (Project JOINTS) that used the Institute for Healthcare Improvement’s (IHI) Rapid Spread Network (RSN) to promote adoption of evidence-based surgical site infection (SSI) prevention practices.

Methods:

We analyzed rates of SSI among Medicare beneficiaries undergoing hip and knee arthroplasty during pre-intervention (5/2010–4/2011) and post-intervention (11/2011–9/2013) periods in 5 states included in a multi-state trial of the Project JOINTS campaign and 5 matched comparison states. We used generalized-linear mixed effects models and a difference-in-differences approach to estimate changes in SSI outcomes.

Results:

125,070 patients underwent hip arthroplasty in 405 hospitals in intervention states, compared with 131,787 in 525 hospitals in comparison states. 170,663 patients underwent knee arthroplasty in 397 hospitals in intervention states, compared with 196,064 in 518 hospitals in comparison states. Post-campaign, patients in intervention states had a 15% lower odds of developing hip arthroplasty SSIs (OR=0.85, 95% confidence interval 0.75–0.96, p=0.01) and a 12% lower odds of knee arthroplasty SSIs than patients in comparison states (OR=0.88, CI 0.78–0.99, p=0.04).

Conclusions:

An IHI RSN campaign to promote adoption of an SSI prevention bundle reduced SSIs following hip and knee arthroplasty.

In 2011, the Institute for Healthcare Improvement (IHI) launched Project JOINTS (Joining Organizations IN Tackling SSIs),¹ a quality improvement (QI) campaign to disseminate a prevention “bundle” aimed at reducing surgical site infection (SSI) following hip and knee arthroplasty. Project JOINTS was based on the spread theory, model, and methods used in IHI’s previous 100,000 Lives and 5 Million Lives Campaigns, which aimed to accelerate hospital adoption of evidence-based practices to improve patient safety at state and national levels.^2,3 This spread theory and model were adapted from previously published political campaign strategies, and the methods were predicated on spread of evidence-based practices beyond hospitals directly receiving the intervention.^4,5

In states that participated in Project JOINTS, IHI leveraged a network of state “nodes” (mainly state hospital associations and Quality Improvement Organizations supported by the Centers for Medicare & Medicaid Services). This so-called “Rapid Spread Network” (RSN) was deployed to disseminate basic quality improvement tools and to catalyze adoption of hip and knee arthroplasty infection preventions practices.

Hospitals that agreed to participate were encouraged to join campaign activities remotely (primarily via a webinar call series and an email listserv) or in person (“town halls” and site visits) and to use educational materials posted on the campaign website. IHI developed a variety of resources for participating hospitals, including: a how-to guide, improvement tools, a patient/family one-page overview, measurement tools, phone-based office hours, and a business case analysis. Participating hospitals were also provided with instruction on using a multifaceted QI approach developed by IHI to motivate and support adoption of the SSI prevention bundle.⁶

The SSI prevention bundle included five evidence-based practices for all adult patients undergoing hip or knee arthroplasty.^7,8 These were:

1. Pre-operative screening for Staphylococcus aureus carriage (both methicillin-sensitive S. aureus and methicillin-resistant S. aureus) and decolonization of carriers with 5 days of twice-daily intranasal mupirocin,^9,10

2. Bathing or showering with 4% chlorhexidine gluconate for 3 days prior to surgery,^11,12

3. Use of an alcohol-containing antiseptic agent for perioperative skin preparation,¹³

4. Avoidance of razors for hair removal,¹⁴ and

5. Appropriate timing of prophylactic antibiotics (including the addition of vancomycin in patients with known MRSA colonization).¹⁵

A prior study showed that Project JOINTS increased adoption of the prevention bundle by 23%.¹⁶ Based on the known effectiveness of the bundle elements in reducing the risk of SSI following hip and knee arthroplasty,^7–15 we hypothesized that campaign states would also show reductions in SSI. This paper presents an analysis of the change in SSI outcomes in 5 states that participated in Project JOINTS interventions beginning in May 2011. We compared these states to 5 matched comparison states that did not participate in the QI campaign.

Methods

Study Population

Using Medicare Provider and Analysis Review (MedPAR) Part A Inpatient Claims data, we studied two clinical cohorts (hip arthroplasty and knee arthroplasty). The hip arthroplasty cohort included fee-for-service Medicare beneficiaries who underwent primary hip arthroplasty (ICD-9 code 81.51 and 81.52) between May 2010 and September 2013, and the knee arthroplasty cohort included fee-for-service Medicare beneficiaries who underwent primary knee arthroplasty (ICD-9 code 81.54) between May 2010 and September 2013.¹⁷

Intervention states included Arkansas, Colorado, Michigan, New York, and Tennessee. Prior to intervention, a comparison state was selected for each intervention state from a list of 18 states that had previously participated in the IHI Rapid Spread Network and expressed interest in participating in Project JOINTS. Matching was based on Medicare data for each state summarizing the number of hospitals performing orthopedic surgery and their orthopedic surgical volume. This was done to balance the populations in intervention and control states and to reduce bias that might arise because of volume-outcome relationships. The five comparison states were Connecticut, Illinois, Missouri, Texas, and Washington. These states did not receive the campaign interventions and thus served as a control.

We included data from all hospitals in the intervention and comparison states and did not limit analysis to hospitals that formally committed to participating in Project JOINTS campaign activities via a letter of commitment from hospital leadership (intention-to-treat analysis). This was decided a priori because of the likelihood of the spread of evidence-based practices beyond participating hospitals in the same state (so called “spillover effect”), as seen in prior IHI campaigns.^2,3 Indeed, such spread was encouraged by the state nodes and is considered desirable when conducting a campaign. To support this spread, the Project JOINTS team designed interventions to take advantage of existing professional and informal social networks within and across organizations. It was predicted that these networks would promote some spread of best practices from hospitals that signed up for Project JOINTS to those that did not.

We did not believe that there would be spread to states other than the intervention states because: 1) the selected states were geographically separated from one another; 2) only the intervention states were invited to participate in IHI’s campaign-associated activities; 3) only providers in intervention states were granted access to resources and tools on a dedicated Project JOINTS website; and 4) the professional societies supporting Project JOINTS agreed to disseminate guidance on practices included in the campaign only within the intervention states. As confirmation, we note that the randomized trial found no evidence of spillover of the intervention beyond the intervention states, while noting some uptake in non-participating hospitals in intervention states.¹⁶

Identifying SSI Events

Following a hip arthroplasty or knee arthroplasty procedure, we analyzed post-operative Medicare claims data for International Classification of Diseases, Ninth Revision (ICD-9) codes suggestive of an SSI within 90 days of the surgical procedure. We selected this surveillance window to be consistent with the current CDC guidelines for SSI surveillance.¹⁸ Prior work has demonstrated that ICD-9 codes 996.66, 996.67, and 998.59 outperform traditional hospital-based surveillance and can be used to compare SSI outcomes across hospitals.^19–21 Using linkers found in the MedPAR claims data, we analyzed data both from the surgical admission and any readmissions to an acute care hospital in the 90-day surveillance window.

Main Analysis

We defined three study periods. 1) pre-intervention period (12 months) from May 2010 through April 2011; 2) intervention (implementation) period (6 months) -- a pre-specified period from May through October 2011, during which hospitals were exposed to a series of Project JOINTS campaign activities; 3) post-intervention period (23 months) from November 2011 through September 2013. We excluded procedures from the last three months of 2013, as we did not have 2014 data to assess the full 90 day post-operative claims data for these procedures.

For the pre-intervention and the post-intervention study periods, we calculated state-level SSI rates in the intervention and the comparison states, excluding data from the 6-month intervention (implementation) period. This was done separately for hip and knee arthroplasty.

We then used a generalized-linear mixed effects model to predict the outcome of SSI. In this model, we adjusted for fixed effects at the patient level to ensure comparison of similar populations. These fixed effects included patient age, gender, and each of the comorbidities in the Elixhauser comorbidity score handled dichotomously as yes/no variables.^22,23 We also adjusted for random intercepts (random effects) for hospital and state to account for clustering,^20,21 study arm (intervention versus comparison state), study period (post versus pre-intervention), and an interaction term between the study arm and the study period. This interaction term provided the odds of a SSI in the post vs. pre-intervention period comparing outcomes in intervention vs. comparison states. This is a difference-in-differences analysis, in which an odds ratio <1 reflects a larger decline in the odds of a SSI in the intervention states relative to the comparison states following the intervention.

Finally, we used a linear regression model to test the parallel trends assumption in the pre-intervention period (May 2010 through April 2011). This was done to assess whether the slope of the monthly SSI rates was different across the 12 months in the intervention versus control states, so as not to attribute a lower SSI rate in the post-intervention period to a trend that was occurring prior to the Project JOINTS interventions.^24,25 We modeled the SSI rate, including terms for study arm (intervention versus comparison state), time (month), and an interaction term between the study arm and time. This interaction term assessed whether the slope over time was significantly different when comparing the trend in intervention versus comparison states.

Sensitivity Analysis

To explore the impact on hospitals in intervention states that signed up to participate, we performed a sensitivity analysis. We modified the difference-in-differences analysis, limiting the intervention arm to hospitals that signed up to participate in the Project JOINTS campaign activities via a letter of commitment from hospital leadership during the intervention period. This allowed us to calculate the odds of a SSI in the post vs. pre-intervention period comparing outcomes in directly participating hospitals in intervention states vs. all hospitals in comparison states.

Results

Study Population

From May 2010 through September 2013, there were 125,070 Medicare beneficiaries who underwent primary hip arthroplasty in 405 hospitals in the 5 intervention states, and 131,787 Medicare beneficiaries who underwent primary hip arthroplasty in 525 hospitals in the 5 comparison states. In addition, there were 170,663 Medicare beneficiaries who underwent primary knee arthroplasty in 397 hospitals in the 5 intervention states, and 196,064 Medicare beneficiaries who underwent primary knee arthroplasty in 518 hospitals in the 5 comparison states.

Table 1 shows similar patient characteristics between Medicare beneficiaries who underwent hip arthroplasty (1A) and knee arthroplasty (1B) in intervention versus comparison states. This table also includes patient characteristics from hospitals that signed up to participate in the Project JOINTS campaign activities during the intervention period.

Table 1A:

Patient Characteristics for Hip Arthroplasty Procedures (May 2010 – September 2013)

	Hip Arthroplasty
	Comparison States (N = 131,787)	Intervention States (N = 125,070)	Directly Participating Hospitals in Intervention States (N = 62,595)
Age, Mean (SD)	76.3 (10.1)	75.6 (10.2)	75.7 (10.3)
Female, %	65.5%	65.1%	65.3%
Comorbidities
Congestive heart failure, %	9.8%	8.9%	9.2%
Diabetes (uncomplicated and complicated), %	18.9%	18.5%	19.0%
Obesity, %	9.3%	9.1%	8.9%
Peripheral vascular disease, %	4.5%	4.3%	4.2%
Renal failure, %	10.6%	9.8%	9.8%
Length of Stay for Surgical Admission, Mean (SD)	4.4 (3.0)	4.5 (3.7)	4.4 (3.7)

Table 1B:

Patient Characteristics for Knee Arthroplasty Procedures (May 2010 – September 2013)

	Knee Arthroplasty
	Comparison States (N = 196,064)	Intervention States (N = 170,663)	Directly Participating Hospitals in Intervention States (N = 88,523)
Age, Mean (SD)	72.1 (7.8)	71.8 (8.0)	71.7 (8.1)
Female, %	64.6%	65.6%	65.9%
Comorbidities
Congestive heart failure, %	4.0%	3.9%	3.9%
Diabetes (uncomplicated and complicated), %	24.2%	23.8%	24.3%
Obesity, %	18.7%	17.7%	17.3%
Peripheral vascular disease, %	2.5%	2.5%	2.5%
Renal failure, %	6.3%	6.1%	6.0%
Length of Stay for Surgical Admission, Mean (SD)	3.3 (1.4)	3.3 (1.6)	3.2 (1.6)

Project JOINTS enrolled 218 hospitals in the 5 intervention states, 171 (78%) by the May 2011 launch and 193 (89%) by the end of the intervention period. There were 25 additional hospitals (11%) that enrolled from November 2011 through May 2012. For the purpose of the sensitivity analysis focused on directly participating hospitals, we included the 193 hospitals in the 5 intervention states that enrolled during the intervention period (May-October 2011).

Overall, directly participating hospitals performed 50% of the hip arthroplasty procedures in intervention states in both the pre-intervention and post-intervention periods. For knee arthroplasty, directly participating hospitals performed 52% of the procedures in intervention states in both the pre-intervention and post-intervention periods.

Claims-Based SSI Rates

Table 2 shows the number of procedures, the number of SSIs, and the SSI rate for the intervention and comparison states, based on Medicare claims. These data are broken down by study period (pre-intervention, post-intervention) and by procedure type (primary hip arthroplasty, primary knee arthroplasty).

Table 2:

Claims-Based SSI Rates in the Pre-Intervention and Post-Intervention Periods

	Pre-Intervention (5/2010–4/2011)			Post-Intervention (11/2011–9/2013)
	Procedures	Infection	SSI Rate*	Procedures	Infection	SSI Rate*
Hip Arthroplasty
Intervention States	35,930	712	1.98%	71,352	1,172	1.64%
Comparison States	38,355	837	2.18%	74,321	1,590	2.14%
Knee Arthroplasty
Intervention States	50,111	827	1.65%	96,556	1,257	1.30%
Comparison States	58,682	960	1.64%	108,692	1,577	1.45%

^*Based on SSI codes (996.66, 996.67, and 998.59) in Medicare Part A claims

Figures 1 plots the SSI rates by month for hip arthroplasty across the three study periods. Figure 2 plots the SSI rates by month for knee arthroplasty across the three study periods. In both figures, we have added lines indicating the mean SSI rate in the pre-intervention and post-intervention periods for intervention and comparison states.

Figure 1:

Claims-Based SSI Rates for Hip Arthroplasty Across Study Periods in Intervention and Comparison States

Figure 2:

Claims-Based SSI Rates for Knee Arthroplasty Across Study Periods in Intervention and Comparison States

For hip arthroplasty in the intervention states, the mean SSI rates in the pre-intervention and post-intervention periods were 1.98% and 1.64%, respectively. In the comparison states, the mean SSI rates in the pre-intervention and post-intervention periods were 2.18% and 2.14%, respectively.

For knee arthroplasty in the intervention states, the mean SSI rates in the pre-intervention and post-intervention periods were 1.65% and 1.30%, respectively. In the comparison states, the mean SSI rates in the pre-intervention and post-intervention periods were 1.64% and 1.45%, respectively.

For both procedures, the outcomes in individual intervention and comparison states are presented in an appendix, along with an indication of which states were selected as pairs when selecting intervention and comparison states.

Difference-in-Differences Analysis

In our difference-in-differences analysis comparing the risk-adjusted odds of developing a SSI following hip arthroplasty in the post-intervention period compared with the pre-intervention period, we found a 15% greater decline in the odds of a SSI in intervention states relative to comparison states (Odds ratio [OR] 0.85, 95% confidence interval [CI] 0.75 to 0.96, p = 0.01). For knee arthroplasty, the risk-adjusted model found a 12% greater decline in the odds of a SSI in intervention states relative to comparison states (OR 0.88, 95% CI 0.78 to 0.99, p = 0.04).

In our parallel trends analysis, the interaction term showed no significant difference in slope across the 12 months prior to intervention for either hip arthroplasty (p = 0.91) or knee arthroplasty (p = 0.27). This verified that the SSI rate trends were similar in the intervention and comparison states for both procedures prior to the Project JOINTS interventions.

Sensitivity Analyses

When limiting the difference-in-differences analysis to directly participating hospitals in intervention states versus all hospitals in comparison states, we found a 23% greater decline in the risk-adjusted SSI rate for hip arthroplasty in intervention states relative to comparison states (OR 0.77, 95% CI 0.66–0.90, p < 0.01). We did not detect a difference for knee arthroplasty (OR 0.97, 95% CI 0.84–1.12, p = 0.69).

In the directly participating hospitals in intervention states, the mean SSI rates for hip arthroplasty were 2.19% pre-intervention and 1.63% post-intervention. For knee arthroplasty, the mean SSI rates were 1.67% pre-intervention and 1.45% post-intervention.

Discussion

We found that states participating in a QI campaign, leveraging IHI’s Rapid Spread Network and campaign methodology, had a significant reduction in SSIs following hip and knee arthroplasty, compared to a set of matched comparison states that were not participating in the QI campaign. These findings, along with a previous study demonstrating improved adherence to a “bundle” of evidence-based practices known to reduce SSI,¹⁶ provide evidence that a well-conducted QI campaign, including a strong spread network infrastructure and multi-faceted dissemination vehicles, can significantly impact patient outcomes on a large scale.

Our results are important because the evidence for the effectiveness of similar campaigns has been inconclusive and controversial. For example, a comparison of expected versus observed hospital mortality before and after the 100,000 Lives Campaign suggested that a reduction of more than 100,000 deaths was achieved. However, while this Campaign generated considerable enthusiasm, catalyzed major national improvement initiatives in the US and globally, and “changed the conversation” about patient safety, the evaluation was limited because it did not include measures of adoption of recommended practices and did not include a control group to assess impact on associated outcomes.

The strategy for Project JOINTS was not only to publicize evidence-based elements of the SSI prevention bundle that were not yet in common practice, but also to provide orthopedic practices and hospitals with easy, free access to a campaign that provided practical tools and methods to help with adoption and implementation. Our assessment of Project JOINTS suggests that the IHI campaign methodology, including concerted, multi-faceted dissemination of materials, engagement of state quality improvement organizations and state hospital associations, and support from relevant professional societies, can successfully accelerate adoption and implementation of evidence-based practices and improve outcomes. This is important given longstanding evidence of delays in translating proven strategies from clinical trials into practice.²⁶

It should be noted that our study design did not allow us to identify which components of the prevention bundle were most highly associated with the reduction in SSI rates at the patient or hospital level. We were also unable to definitively attribute potential indirect effects that the campaign may have had on hospitals not directly enrolled in Project JOINTS. The theory of the IHI spread network is that direct participation is not a requirement. Instead, non-participating organizations and professionals change their practices as they learn about or observe changes in practice of participating organizations and professionals.

To analyze the impact of these expected statewide practice changes on SSI events, we compared outcomes in Medicare beneficiaries, as these patients’ claims data are accessible via a large, centralized database with good state penetration. We do not have information, however, on the percent of the population covered by Medicare in each of the intervention and comparison states.

It is also important to note that our analyses relied on coding indicative of SSI using Medicare claims as a proxy for SSI events. While not all cases identified by claims would be expected to have a chart-confirmed SSI, previous nationally-validated studies have shown that claims-based surveillance identifies a sizeable proportion of cases missed by traditional surveillance and that the SSI confirmation rate for records flagged by claims is highly consistent across hospitals.^19–21 This is therefore a validated metric for comparing SSI performance, understanding the limitation that the aggregate codes are unable to differentiate superficial SSIs from deep and organ/space SSIs, and concerns about the sensitivity and specificity of billing codes used to identify SSI events.^27,28 It should be noted, though, that these same studies highlight hip and knee arthroplasty as the procedures with the most data to support the use of claims-based surveillance, and we have no reason to suspect variability in the performance of these codes in the intervention versus comparison states in our study.

As for the observation that the decline in the risk-adjusted SSI rate for knee arthroplasty was no longer significantly greater in the sensitivity analysis comparing directly participating hospitals in intervention states versus all hospitals in comparison states, we have a few hypotheses why this might be. First, it is possible that hospitals that enrolled to participate in Project JOINTS in the intervention states were already working to reduce SSI rates prior to enrollment. Second, some of the hospitals that participated in Project JOINTS activities enrolled after the intervention period, such that impacts on these later participating hospitals would only be seen when looking at the data in our main analysis. Finally, the SSI rates for knee arthroplasty were lower than for hip arthroplasty, so there was less opportunity for improvement.

As for the observation that the decline in risk-adjusted SSI rate for hip arthroplasty was greater in the sensitivity analysis comparing directly participating hospitals in intervention states versus all hospitals in comparison states, it is worth pointing out that directly participating hospitals in the intervention states had a higher SSI rate in the pre-intervention period than non-participating hospitals in the intervention states. This may have been why these hospitals enrolled to participate.

Finally, data availability did not allow us to look at sustainability of the intervention beyond 23 months following Project JOINTS. While data from 2014 and beyond could help to look at sustainability, the data presented in Figures 1 and 2 do not show a loss of impact over time based on a rising slope, although not tested statistically. The sustainability of this quality improvement work is an important area for future study.

In conclusion, this evaluation of the Project JOINTS campaign demonstrated a significant reduction in SSIs in intervention states following the campaign in comparison with non-participating comparison states. The IHI Rapid Spread Network campaign approach provides a promising platform for large-scale spread and adoption of evidence-based practices, with improved patient outcomes.