Evaluating implementation of a rapid response team: considering alternative outcome measures

Abstract

Objective

Determine the prolonged effect of rapid response team (RRT) implementation on failure to rescue (FTR).

Design

Longitudinal study of institutional performance with control charts and Bayesian change point (BCP) analysis.

Setting

Two academic hospitals in Midwest, USA.

Participants

All inpatients discharged between 1 September 2005 and 31 December 2010.

Intervention

Implementation of an RRT serving the Mayo Clinic Rochester system was phased in for all inpatient services beginning in September 2006 and was completed in February 2008.

Main Outcome Measure

Modified version of the AHRQ FTR measure, which identifies hospital mortalities among medical and surgical patients with specified in-hospital complications.

Results

A decrease in FTR, as well as an increase in the unplanned ICU transfer rate, occurred in the second-year post-RRT implementation coinciding with an increase in RRT calls per month. No significant decreases were observed pre- and post-implementation for cardiopulmonary resuscitation events or overall mortality. A significant decrease in mortality among non-ICU discharges was identified by control charts, although this finding was not detected by BCP or pre- vs. post-analyses.

Conclusions

Reduction in the FTR rate was associated with a substantial increase in the number of RRT calls. Effects of RRT may not be seen until RRT calls reach a sufficient threshold. FTR rate may be better at capturing the effect of RRT implementation than the rate of cardiac arrests. These results support prior reports that short-term studies may underestimate the impact of RRT systems, and support the need for ongoing monitoring and assessment of outcomes to facilitate best resource utilization.

Introduction

The theoretical value of rapid response teams (RRT), a designated group of healthcare providers with critical care expertise who can assemble quickly to deliver care at the bedside of deteriorating patients, has been recently extolled [1], even though the evidence supporting tangible positive effects is far from convincing. An earlier systematic review reported inconclusive evidence regarding improvement to hospital mortality and cardiac arrest rates. [2] A more recent systematic review has given more evidence to improved outcomes while noting this is not realized in all cases [3]. Predominantly, studies showing positive effects have come from single center studies [3–10], while a large multi-center randomized trial failed to show improved outcomes [11].

RRTs are but one aspect of the rapid response system (RRS), and are commonly referred to as the efferent arm. The impact of this team on outcomes is highly dependent on the timely detection of deterioration and activation, a process commonly referred to as the afferent limb of the RRS. Hence, the performance of a response team as an isolated component of the RRS may be insufficient to demonstrate improved outcomes [12]. First, a culture of acceptance is important to the overall success of the RRS. The act of calling RRT to assist with a deteriorating patient may be viewed by primary teams as an acknowledgement of inadequacy on their part. This scenario has been postulated as a potential explanation of the mixed results in the literature regarding RRT effectiveness. Secondly, RRT may require sufficient utilization to enable improved outcomes to be detected statistically [13]. Thirdly, improved outcomes may not occur until the RRT system has been in place for an extended period of time, enough time for the process to have stabilized. Finally, a definitive outcome measure has yet to be identified which adequately measures the effectiveness of RRT implementation.

Prior studies focused on hospital mortality, cardiac arrest outside the intensive care unit (ICU) and unplanned transfers to the ICU [3–10]. It is not clear that these measures are the most appropriate outcome measurements to evaluate RRTs. Estimates of the effectiveness of RRTs may be too diluted by unpreventable deaths' impact on total hospital mortality. Preventing cardiac arrests outside the ICU only addresses some of the conditions for which RRT is called. In addition, cardiac arrest rates are confounded by palliative care practices and how effectively ‘do not resuscitate orders’ are applied for end-of-life patients. There also has been variability in the definition and measurement of cardiac arrest, and even with careful definition, accurate rates may be difficult to determine retrospectively from hospital records [14]. Finally, unplanned transfer to ICU is a surrogate outcome measure where the most desirable clinical outcome is unclear. Effective RRTs might reduce ICU transfers, but they could also result in an increased transfer rate by more accurately determining which patients are deteriorating and need more intensive treatment. The review by Winters et al. is arguably the most convincing, while not conclusive, evidence to date supporting the benefits of RRTs [3]. Yet, this review suggests no alternative outcome measures that may be more suitable.

A different outcome that holds potential value for measuring the effect of RRT implementation is the original failure to rescue (FTR) patient safety indicator measure defined by the Agency for Healthcare Research and Quality (AHRQ) [15]. This measure (which was renamed ‘Death among surgical inpatients with serious treatable complications’) was originally designed to calculate the hospital mortality rate among medical and surgical patients having specific in-hospital complications: acute renal failure, pneumonia, deep vein thrombosis or pulmonary embolism (DVT/PE), sepsis, shock and/or cardiac arrest (SCA) and gastro-intestinal (GI) bleeding. Once the complication occurs, having an RRT available could affect the outcome of many of these patients. In addition, examining individually each of the six complications may further enhance the usefulness of the measure as the RRT might be expected to impact FTR differently for each condition. The aim of our study was to investigate the use of the AHRQ FTR measure for quantifying the prolonged effect of RRT implementation when compared with other measures. We hypothesized that after RRT implementation the FTR measure would decrease as patients with deterioration on general floors were more rapidly recognized and appropriately addressed.

Methods

Study setting and population

This study took place at Mayo Clinic, Rochester, MN, which includes two acute care hospitals: Saint Mary's Hospital (SMH) and Rochester Methodist Hospital (RMH), plus the inpatient Mayo Clinic Psychiatry and Psychology Treatment Center. The study population included all inpatients discharged between 1 September 2005 and 31 December 2010. Beginning in September 2006, implementation of an RRT serving the Mayo Clinic Rochester system was phased in for all inpatient services. Implementation began in SMH and was completed in RMH in February 2008. On-going education about recognition and response is provided to hospital clinical staff. For our study, pre-implementation covers the 12 months from September 2005 through August 2006 where we had resuscitation calls data available, while full post-implementation covers the time from March 2008 through December 2010. In accordance with Minnesota statutes, all patients denying research authorization to their medical records were excluded from the assessment. This study was approved by the Mayo Institutional Review Board.

Rapid response system

The efferent aspect of the RRS at Mayo Clinic Rochester hospitals is a multidisciplinary team including a critical care nurse, critical care fellow and respiratory therapist. They are supervised 24/7 by an in-house attending level intensivist. Any care provider may activate the rapid response system based on concern or physiologically based criteria. The system has been in place for >6 years and provides coverage for two tertiary hospitals. Activation rates average between 50/1000 and 70/1000 discharges. Approximately 60% of RRT calls are among medical patients.

Data sources

All data utilized in this study were retrospectively gathered from administrative databases. Patient discharge information, which also includes procedure and diagnosis codes, was obtained from billing sources. Details on ICU transfers were identified from the bed control systems designed to track the movements of patients during hospitalization. Information on RRT and cardiopulmonary arrests outside the ICUs, as measured by resuscitation events, was collected from clinical registries that capture calls for both of these events. Pediatric cases account for 2.5% of RRT and resuscitation events.

Outcomes

We used a modified version of the AHRQ FTR measure Version 2.1 Revision 1 [16]. This AHRQ FTR measure identifies hospital mortalities among medical and surgical patients having at least one of the specified in-hospital complications. The AHRQ measure used an algorithm to determine the inclusion and exclusion criteria based primarily on secondary diagnosis codes. The exclusionary criteria were designed to eliminate cases that were most likely to have complications present on admission. Versions prior to Version 3.1 of the algorithm were designed before the general availability of present on admission information. We modified the Version 2.1 FTR algorithm to take advantage of present-on-admission coding [17] using only acquired codes as input to identify complications and did not use the exclusionary criteria of the original measure. Our previous analysis showed that medical patients with FTR conditions not present on admission had significantly higher FTR rates than patients with those conditions present on admission [16]. Using the current AHRQ measure (PSI #4. Death among Surgical Inpatients with Serious Treatable Complications), which focuses only on surgical patients would ignore the majority of cases receiving RRT activations.

All transfers from the hospital floor to an ICU were identified. These transfers to the ICU were considered to be unplanned only if no surgical procedure occurred the day of the transfer. The unit of measure was the number of unplanned ICU transfers per 1000 hospital floor days. Resuscitation events outside the ICU were the subset of emergency response calls, which were initiated for cardiopulmonary arrest (requiring cardiopulmonary resuscitation or defibrillation) or acute respiratory compromise (requiring intubation or bag/mask ventilation). Overall hospital mortality percentage was obtained per month, as well as the subset of hospital deaths occurring outside the ICU.

Statistical analysis

Three separate and independent analyses were performed on each outcome measure at the institution level to analyze effects over time. Two of these approaches, control charts and Bayesian change point (BCP) analysis, allow the identification of important changes in the measure of interest. The third analysis used pre-defined time periods to statistically test the measures. We tested the measures between the pre-implementation (baseline) and full post-implementation time periods as described above. The rollout period was not analyzed in this approach.

Control charts were plotted by discharge month. To provide a stable estimate, initial control chart means and limits were based on the first 18 months. In each control chart, a pattern of eight or more consecutive data points occurring above or below the overall mean was considered to signify a significant change in process [18–21]. After a change was identified, new control chart means and limits were calculated. The total number of RRT calls by month were determined and overlaid on the control charts.

BCP models calculate probability distributions of change point time and magnitude in change of FTR. This approach complements control charts since change point analyses can detect smaller changes potentially missed by control charts. Control charts determine fixed locations of a shift in means, while BCP analysis methods use partitioning of the time interval and calculate the probability of change at each location. The partitioning results in a sequence of variables that indicate whether a change point has occurred with probability P. The Bayesian approach assumes prior distributions of the means in each block of the partition, then estimates the probability of a change point and updates the posterior means based on the data, probabilities, and blocks of the partitions [22–25]. The choice of a cut-off in the posterior probabilities can be subjective. In our situation a cut-off value of 0.2 was chosen which corresponds to the highest 5% of the probabilities.

Monthly outcome measures for the pre-defined pre-implementation vs. full post-implementation time periods were compared using the Wilcoxon rank sum test. P-values of 0.05 or less were considered to be significant. All analyses were performed using the statistical software packages R version 2.15.1 [23] and SAS 9.2.

Results

Throughout the entire 6 years reported, our institutions averaged 5288 discharges per calendar month. The number of cases qualifying for FTR inclusion averaged 207 cases per month and was fairly consistent over time with an increase from 181 cases per month before the national mandate to provide present-on-admission coding in October 2007 to 227 cases per month after. The overall FTR control chart, along with the overlay of the RRT calls per month, is depicted in Fig. 1. The initial mean monthly FTR rate based on the control chart was 17.8%. Both the control chart rules and BCP analysis (probability = 0.59) indicated that a significant drop in the FTR rate occurs ∼12 months after full RRT implementation (see Fig. 2). The post-shift mean FTR rate from the control chart was 13.8%. This change coincided with an increase in the monthly count of RRT calls. The BCP analysis identified two shifts and thus split the time period into three blocks: before 12/2006, 12/2006 to 2/2009, and after 2/2009. The first shift in FTR occurs shortly after implementation began. The time period from 12/2006 to 2/2009 exhibits non-stationary behavior, namely mean and variance are not constant, possibly slowly decreasing over time. The period after 2/2009 corresponds to a stationary shift to lower FTR as also indicated by the control chart, 1 year post-implementation. There was a statistically significant difference in the FTR rate when comparing the pre-implementation period to the full post-implementation period (P = 0.016) (Supplementary data, Appendix Table S1). Complication-specific FTR rates from control charts are also provided in the Supplementary data, Appendix. A downward shift occurred for SCA 10 months after RRT implementation. A downward shift of the renal failure FTR rate occurred 12 months after full RRT implementation. An increase in the FTR rate occurred for GI bleeding, which roughly coincided with full RRT implementation. No changes were noted in the FTR rates for pneumonia, DVT/PE or sepsis.

Figure 1.

Monthly FTR rate control chart and monthly RRT calls. The FTR rate is presented with a control chart with limits based on the first 20 monthly data points. New control limits starting in March 2009 were set based on the control chart rule of eight consecutive data points below the process mean. Numbers of calls per month for the RRT are overlaid beginning with the start of RRT implementation in September 2006. Implementation was completed in February 2008.

Figure 2.

BCP analysis of FTR rate. The bottom panel of this figure shows the estimated probabilities that a change point occurred. For example, there is a probability of 0.6 on 2/2009 that a change occurred as also evidenced by the drop in posterior means in the top panel of the figure. The BCP ‘bumps’ at June (2007) and February (2009) are mirrored by apparent changes in the FTR line as shown in Fig. 1.

When comparing the complication-specific FTR rates pre- and full post-RRT implementation, we noted that both renal failure (P = 0.007) and SCA (P < 0.001) had significantly reduced rates in the full post-implementation period. There was some evidence, although not statistically significant, for a decrease in sepsis FTR between pre- and full post-implementation periods (P = 0.064). There was also some evidence, although not statistically significant, of an increase in FTR for GI bleeding (P = 0.062). No significant change was found in pre- vs. post-implementation rates of DVT/PE (P = 0.831) or pneumonia (P = 0.861).

The unplanned ICU transfer rate is depicted in Fig. 3. The control chart rules indicated that a significant upward shift occurred 18 months after the start of the post-implementation period, which appears to coincide with increasing RRT calls. The initial mean of the unplanned ICU transfer rate based on the control chart was 13.7 transfers per 1000 floor days. After the shift this increased to 15.2 transfers per 1000 floor days. Four BCP changes were ≥0.2, one at the end of 2005 and the other three occurring the month of, and the month on either side of, the shift identified in the control chart. Focusing on the pre- vs. full post-implementation periods, the unplanned ICU transfer rate increased significantly (P < 0.001).

Figure 3.

Monthly Unplanned ICU transfer control chart and monthly RRT calls. The rate of unplanned ICU transfers per 1000 floor days is presented with a control chart with an initial process mean based on the first 20 monthly data points. New control limits starting in June 2009 were set based on the control chart rule of eight consecutive data points above the initial process mean. Numbers of calls per month for the RRT are overlaid beginning with the start of RRT implementation in September 2006. Implementation was completed in February 2008.

Figure 4 depicts the rate of resuscitation events over the study timeframe. No changes were detected through either control chart or BCP analyses. The rates of events per 1000 discharges in the pre- vs. full post-implementation time periods were not statistically different (P = 0.480). Furthermore, no changes were seen over time separately in either type of resuscitation event; cardiopulmonary arrest or acute respiratory compromise.

Figure 4.

Monthly resuscitation events control chart. The rate of resuscitation events per 1000 discharges is presented with a control chart with an initial process mean based on the first 20 monthly data points. There was no evidence for a change in rate or control limits. Start of RRT implementation began in September 2006 and was completed in February 2008.

Overall hospital mortality in the pre-implementation period was 1.5% compared with 1.6% in the full post-implementation period (P = 0.299). The control chart depicting monthly percent of discharges with a death outside the ICU is displayed in Fig. 5. Control chart rules indicated a change around 15 months post-implementation; however, BCP indicated that there were no detectable changes during the time period. This rate was 0.6% in both pre- and full post-implementation periods (P = 0.970).

Figure 5.

Monthly percent of discharges with a death outside the ICU. The mortality percentage for deaths outside the ICU is presented with a control chart with an initial process mean based on the first 20 monthly data points. New control limits starting in June 2009 were set based on the control chart rule of eight consecutive data points below the initial process mean. Start of RRT implementation began in September 2006 and was completed in February 2008.

Discussion

None of our outcome measures had major changes immediately after the RRT was implemented throughout the hospital. However, results from this analysis revealed a decrease in our FTR measure as well as an increase in the unplanned ICU transfer rate occurring in the second-year post-implementation of RRT. Two complementary statistical techniques, control charts and BCP analysis, were used to objectively detect when the shift occurred. Both the shift in FTR and transfer rates correspond with an increase in the number of RRT calls per month. These changes occurred during educational efforts among our nursing staff to address sepsis response and reinforce the calling criteria and the value of the RRT. Anecdotally, these changes also coincided with efforts among resident-training directors to encourage the acceptance and use of RRT for patients with questionable vital signs. No significant decreases were observed between pre- and post-implementation for resuscitation events or overall mortality. A significant decrease in mortality among non-ICU discharges was identified by control charts, although this finding was not detected by BCP or pre- vs. post-statistical tests.

Furthermore, we saw improved FTR rates for specific complications including acute renal failure and SCA; while seeing no significant change in FTR for DVT/PE, pneumonia and sepsis. An increasing shift was seen in GI bleeding; however, this was suggestive, but not significant, in the pre- vs. full post-secondary analysis. It is expected that if an RRT system is effective with timely interventions for the clinically deteriorating patient, there should be fewer deaths due to shock or cardiac arrest. It is interesting that we found no relationship between RRT implementation and FTR for sepsis and pneumonia, although possibly an absence of shock state reduces risk of death significantly. It is possible a majority of patients with DVT/PE and pneumonia improve with institution of appropriate therapy, and most of these patients do not demonstrate clinical deterioration to the point where activation of the RRS is necessary. The patients with the extreme ends of the spectrum (i.e. ARDS, or saddle embolus resulting in shock states) are relatively uncommon and may represent conditions that are rarely salvageable irrespective of intervention. The positive improvement for FTR in renal failure is an interesting finding that requires further investigation, but one hypothesis might be that life-threatening electrolyte and fluid disturbances that occur with this disease process are highly treatable with acute interventions. Similarly to other studies, we found no change in the rate of cardiac or respiratory resuscitations outside the ICUs [10]. Unlike other studies, our study saw an increasing trend in unplanned ICU transfers [10, 26, 27].

Visualizing the data over time helped shed light regarding the effects of RRT implementation. The upward shift in unplanned ICU transfer rates was not observed until 18 months after RRT implementation. This supports prior hypotheses that effects from RRT implementation will not be immediately noticeable [11]. Furthermore, this shift coincided with a rapid increase in the number of RRT calls, suggesting that there may be a threshold of RRT calls that must be reached in order to detect favorable effects. This highlights the importance of education and acceptance of RRT calls and reinforces the underlying need for a culture change along with increased resources.

Although pediatric patients were included in our study, they represented a small portion of either the RRT calls (2.5%) or FTR eligible patients (3.5%). Pediatric RRT activations are often much different from adult cases and the applicability of our finding to this population is limited.

To our knowledge, we are the first to use the original AHRQ FTR (modified or otherwise) as an outcome measure for evaluating RRTs. This outcome measure may be much better suited to detect change compared with overall mortality rates, as one would not expect RRT to have an effect on all hospital mortalities. Ghaferi et al. [28] used FTR to show that surgical performance across academic centers differed in their capacity to rescue rather than in the rates of surgical complications. One caution with use of our FTR modification is that it relies on present-on-admission coding. Although we have been capturing this data at our institution for over 20 years [17], national reporting in the USA began in October 2007, and questions have been raised about the reliability and accuracy of the coding [29]. Medicare has reported good performance of the measures for identifying a small group of hospital acquired conditions [30]. Our change from internal definition to national definition actually occurred during implementation phase of the RRT. Although the number of patients with conditions labeled as not present on admission which qualified for FTR increased at that time, there was no change in the FTR rate at that time.

While this study offers important contributions to the literature, we must acknowledge several limitations. First, the definition of FTR is not universally agreed upon. In this case, we operationalized FTR as previously defined by AHRQ, which has subsequently redefined it. Even Silber et al. [31], who created the original definition for FTR have proposed changes. Unfortunately, the definition continues to vary leading to different interpretations, even within groups of clinicians at the same institution. The ability to analyze results between institutions is very difficult and invariably creates a proverbial ‘apples to oranges’ comparison. Secondly, RRT was not the only intervention designed to decrease mortality during the timeframe. Like most observational studies of quality improvement interventions, rarely is a single intervention implemented. During this timeframe, our institution also undertook full clinical review of all deaths which identified improvements which we hope contributed to improved hospital mortality. We have tried to address this potential confounding by including the complication-specific FTR rates, which reinforced the observation that implementation of the RRT was related to the change in FTR. The greatest improvement in FTR in absolute terms came from a decrease in the FTR among shock and cardiac arrest patients. Another limitation is that our calculation of FTR relies on diagnosis codes captured for the billing process. We did not review these cases for diagnostic accuracy. Finally, our assumptions may have influenced our conclusions. Specifically, our threshold for indicating an important change under BCP was chosen at the 95th percentile. Other choices may have resulted in identifying different phases. To address this issue, we used the control chart methodology as our principal analysis to identify changes and BCP as a supplementary method, with statistical tests to confirm that the outcome was different after implementation.

Conclusions

It appears in our data that a reduction in the FTR rate is associated with a substantial increase in the number of RRT calls. Effects of RRT may not be seen until RRT calls reach a sufficient threshold. We also observed a significant increase in the rate of unplanned ICU transfers, possibly due to many deteriorating patients seen by the RRT being transferred to the ICU. FTR rate may be better at capturing the effect of RRT implementation than the rate of resuscitation calls. Implementation of an RRT system is a complex clinical project requiring intensive education, staff awareness and resource reallocation efforts. These results support prior reports that short-term studies may underestimate the impact of RRT systems, and support the need for ongoing monitoring and assessment of outcomes to facilitate best resource utilization.

Supplementary material

Supplementary material is available at INTQHC Journal online.

Funding

Funding was provided by the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery.

Conflict of interest

All authors report no relationship or financial interests with any entity that would pose a conflict of interest with the subject matter of this article.